Re: Common elective orthopaedic procedures and their clinical effectiveness: umbrella review of level 1 evidence and the Straw man Fallacy
Dear Editor
We wish to highlight several concerns in relation to “Common elective orthopaedic procedures and their clinical effectiveness: umbrella review of level 1 evidence” by Blom et al.
Whilst the premise of the work, to evaluate orthopaedic practice rigorously, is laudable and one we strongly support, the article provides a very narrow perspective on the evidence. The authors ignore high-quality observational data to arrive at a sensational conclusion: “No strong, high quality evidence base shows that many commonly performed elective orthopaedic procedures are more effective than non-operative alternatives.” This conclusion is not justified and is misleading to patients, healthcare professionals and healthcare commissioners. This study has created unnecessary uncertainty for patients in severe pain who need surgery, having exhausted all alternative options.
1. Absence of Level 1 evidence is not evidence for absence of clinical effectiveness, nor an indication for a randomised controlled trial
The study is designed to critically summarise Level 1 evidence for common orthopaedic procedures. However, the authors have missed the key step necessary for any clinical trial: there must be treatment uncertainty. How was such an important step missed?
Whilst the authors correctly recognise that absence of Level 1 evidence is not evidence for absence of clinical effectiveness, they do not seem to have recognised that neither does it mean that a randomised controlled trial is necessarily needed. For example, one would not consider a trial on antibiotics versus placebo for septicaemia to be necessary, or indeed ethical. The same logic applies to hip and knee joint replacement for osteoarthritis, where the evidence for fundamental efficacy is overwhelming.
A ‘Straw Man Fallacy’ is nicely described at peer review. The authors’ argument is centred on the presence or absence of very specific types of evidence, knowing that readers may conflate this with the more important question of clinical effectiveness. The potential for this misinterpretation has already been realised and, when this article was referred to in The Daily Mail, with the headline: “Bone and joint surgery may be no more useful than physiotherapy or medication, new study suggests”. This is clearly misplaced and potentially harmful logic.
2. Why was observational evidence ignored?
The title of the article is misleading. It suggests to readers that the authors have evaluated the clinical effectiveness of common orthopaedic procedures. In fact, the authors have taken a very selective approach to evaluation, where non-randomised evidence was ignored. It is ironic that the authors have backgrounds in observational research and are part of a team involved with audit of national orthopaedic data. Much of their previous research has extolled the virtue and longevity of joint replacement (1,2).
The authors have ignored the realities of elective orthopaedic practice in the UK where we have among the best developed infrastructure of any specialty to routinely capture outcomes from the patient’s perspective. The NHS PROMs programme has collected data on more than 1,000,000 joint replacement procedures since 2009. The results from 2019-20 showed that 97% of patients reported improvement in their symptoms following hip replacement surgery. The average gain in Oxford Hip Score was 22.1 points and the gain in EQ-5D utility was 0.5 at six months following surgery (3). The large size of these observed effects is not seen with any non-operative intervention, nor through natural fluctuations in symptom severity. Whilst we might choose to develop these complex interventions differently today were we starting anew, there can be no rationale to ignore the observational evidence that already exists. This is supported by relevant guidance on developing and evaluating complex interventions - such as that from the Medical Research Council (incidentally, a highly cited article in the BMJ) (4).
The authors seem to have given little consideration that randomised controlled trials (RCTs) are not always the optimal vehicle for providing the best quality evidence, especially for interventions with excellent long-term outcomes where adverse outcomes can be rare and present late. As the authors are aware, the revision rate following total hip replacement is less than 5% at 10-year follow-up (5).
The potential for a placebo effect from surgery to contribute substantially to any benefit gained forms a large part of the authors’ rationale in this study. Whilst placebo effects can never be entirely ruled out, in this case, the inference is incorrect. Well-conducted RCTs (6–8) have demonstrated that the large clinical benefits experienced by patients after joint replacement persist to medium and long-term follow-up. Apart from understanding mechanisms, placebo-controlled trials are only of value when high levels of uncertainty for effect exist.
Elective orthopaedics has embraced uncertainty around treatment where it exists and an impressive portfolio of randomised trials have been completed or are ongoing (6,7,9–11). Forensic exploration of the evidence base for medical intervention is to be encouraged, but must be done responsibly, without sensationalism and with consideration of the patient as paramount.
yours sincerely
Professor John A Skinner FRCS
Vice President British Orthopaedic Association
Consultant Orthopaedic Surgeon
Stanmore
UK
Mr Tim Wilton FRCS
Medical Director, National Joint Registry of England, Wales, Northern Ireland and the Isle of Man
Consultant Orthopaedic Surgeon
Derby
UK
Professor Fares S Haddad FRCS
Editor, Bone and Joint Journal
Consultant Orthopaedic Surgeon
London
UK
Professor Hamish Simpson FRCS
Editor, Bone and Joint Research Journal
Chair, British Orthopaedic Association Research Committee
Consultant Orthopaedic Surgeon
Edinburgh
UK
Mr Robert Handley FRCS
President, British Orthopaedic Association
Consultant Trauma Surgeon
Oxford
UK
1. Evans JT, Evans JP, Walker RW, Blom AW, Whitehouse MR, Sayers A. How long does a hip replacement last? A systematic review and meta-analysis of case series and national registry reports with more than 15 years of follow-up. Lancet (London, England) [Internet]. 2019;393(10172):647–54. Available from: http://dx.doi.org/10.1016/S0140-6736(18)32531-5
2. Evans JT, Walker RW, Evans JP, Blom AW, Sayers A, Whitehouse MR. How long does a knee replacement last? A systematic review and meta-analysis of case series and national registry reports with more than 15 years of follow-up. Lancet [Internet]. 2019;393(10172):655–63. Available from: http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=med...http://oxfordsfx.hosted.exlibrisgroup.com/oxford?sid=OVID:medline&id=pmi...
3. NHS Digital. Patient Reported Outcome Measures (PROMs) [Internet]. [cited 2021 Jun 23]. Available from: https://digital.nhs.uk/data-and-information/data-tools-and-services/data...
4. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008 Sep;337:a1655.
5. National Joint Registry for England Wales Northern Ireland and the Isle of Man. NJR 17th Annual Report. 2020;(December 2019).
6. Murray DW, MacLennan GS, Breeman S, Dakin HA, Johnston L, Campbell MK, et al. A randomised controlled trial of the clinical effectiveness and cost-effectiveness of different knee prostheses: the Knee Arthroplasty Trial (KAT). Health Technol Assess [Internet]. 2014;18(19):1‐235, vii‐viii. Available from: http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=med...http://oxfordsfx.hosted.exlibrisgroup.com/oxford?sid=OVID:medline&id=pmi...
7. Beard DJ, Davies LJ, Cook JA, MacLennan G, Price A, Kent S, et al. The clinical and cost-effectiveness of total versus partial knee replacement in patients with medial compartment osteoarthritis (TOPKAT): 5-year outcomes of a randomised controlled trial. Lancet [Internet]. 2019;394(10200):746–56. Available from: http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=med...http://oxfordsfx.hosted.exlibrisgroup.com/oxford?sid=OVID:medline&id=pmi...
8. Beard DJ, Price AJ, Cook J, Fitzpatrick R, Carr A, Campbell M, et al. Total or partial knee replacement for medial osteoarthritis? Early results from the TOPKAT trial. Arthrosc - J Arthrosc Relat Surg [Internet]. 2017;33 (10 Sup:e91–2. Available from: http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=eme...http://oxfordsfx.hosted.exlibrisgroup.com/oxford?sid=OVID:embase&id=pmid...
9. Beard D, Rees J, Rombach I, Cooper C, Cook J, Merritt N, et al. The CSAW Study (Can Shoulder Arthroscopy Work?) - a placebo-controlled surgical intervention trial assessing the clinical and cost effectiveness of arthroscopic subacromial decompression for shoulder pain: Study protocol for a randomised controlled trial. Trials [Internet]. 2015 May 9 [cited 2021 Jul 31];16(1):210. Available from: https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-015-0725-y
10. Metcalfe A. A comparison of standard versus robotic total knee replacement [Internet]. [cited 2021 Jul 31]. Available from: https://www.isrctn.com/ISRCTN27624068
11. Anderson DB, Ferreira ML, Harris IA, Davis GA, Stanford R, Beard D, et al. SUcceSS, SUrgery for Spinal Stenosis: Protocol of a randomised, placebo-controlled trial. BMJ Open [Internet]. 2019 Feb 1 [cited 2021 Jul 31];9(2):24944. Available from: http://bmjopen.bmj.com/
Competing interests:
All authors are practising orthopaedic surgeons and work in orthopaedic related organisations as stated. These are the British Orthopaedic Association (5500 members “Supporting surgeons Caring for patients”), National Joint Registry (over 3 million records of hip, knee, shoulder, elbow and ankle replacements) and the Editors of two of the most important trauma & orthopaedic journals. All of the authors have consulted for or received institutional research funding from orthopaedic implant manufacturers in the past. Professor Haddad states that he currently consults for Stryker and Smith & Nephew and that his Hospital receives Institutional research funds from Stryker, S&N, ZimmerBiomet & Corin. All of these are orthopaedic implant manufacturers.
12 August 2021
John A Skinner
Consultant Orthopaedic Surgeon
Mr Tim Wilton Medical Director National Joint Registry, Professor Fares S Haddad, Editor Bone and Joint Journal, Professor Hamish Simpson, Editor Bone and Joint Research, Mr Robert Handley President British Orthopaedic Association
Vice President British Orthopaedic Association
Royal National Orthopaedic Hospital NHS Trust, Stanmore, UK
Rapid Response:
Re: Common elective orthopaedic procedures and their clinical effectiveness: umbrella review of level 1 evidence and the Straw man Fallacy
Dear Editor
We wish to highlight several concerns in relation to “Common elective orthopaedic procedures and their clinical effectiveness: umbrella review of level 1 evidence” by Blom et al.
Whilst the premise of the work, to evaluate orthopaedic practice rigorously, is laudable and one we strongly support, the article provides a very narrow perspective on the evidence. The authors ignore high-quality observational data to arrive at a sensational conclusion: “No strong, high quality evidence base shows that many commonly performed elective orthopaedic procedures are more effective than non-operative alternatives.” This conclusion is not justified and is misleading to patients, healthcare professionals and healthcare commissioners. This study has created unnecessary uncertainty for patients in severe pain who need surgery, having exhausted all alternative options.
1. Absence of Level 1 evidence is not evidence for absence of clinical effectiveness, nor an indication for a randomised controlled trial
The study is designed to critically summarise Level 1 evidence for common orthopaedic procedures. However, the authors have missed the key step necessary for any clinical trial: there must be treatment uncertainty. How was such an important step missed?
Whilst the authors correctly recognise that absence of Level 1 evidence is not evidence for absence of clinical effectiveness, they do not seem to have recognised that neither does it mean that a randomised controlled trial is necessarily needed. For example, one would not consider a trial on antibiotics versus placebo for septicaemia to be necessary, or indeed ethical. The same logic applies to hip and knee joint replacement for osteoarthritis, where the evidence for fundamental efficacy is overwhelming.
A ‘Straw Man Fallacy’ is nicely described at peer review. The authors’ argument is centred on the presence or absence of very specific types of evidence, knowing that readers may conflate this with the more important question of clinical effectiveness. The potential for this misinterpretation has already been realised and, when this article was referred to in The Daily Mail, with the headline: “Bone and joint surgery may be no more useful than physiotherapy or medication, new study suggests”. This is clearly misplaced and potentially harmful logic.
2. Why was observational evidence ignored?
The title of the article is misleading. It suggests to readers that the authors have evaluated the clinical effectiveness of common orthopaedic procedures. In fact, the authors have taken a very selective approach to evaluation, where non-randomised evidence was ignored. It is ironic that the authors have backgrounds in observational research and are part of a team involved with audit of national orthopaedic data. Much of their previous research has extolled the virtue and longevity of joint replacement (1,2).
The authors have ignored the realities of elective orthopaedic practice in the UK where we have among the best developed infrastructure of any specialty to routinely capture outcomes from the patient’s perspective. The NHS PROMs programme has collected data on more than 1,000,000 joint replacement procedures since 2009. The results from 2019-20 showed that 97% of patients reported improvement in their symptoms following hip replacement surgery. The average gain in Oxford Hip Score was 22.1 points and the gain in EQ-5D utility was 0.5 at six months following surgery (3). The large size of these observed effects is not seen with any non-operative intervention, nor through natural fluctuations in symptom severity. Whilst we might choose to develop these complex interventions differently today were we starting anew, there can be no rationale to ignore the observational evidence that already exists. This is supported by relevant guidance on developing and evaluating complex interventions - such as that from the Medical Research Council (incidentally, a highly cited article in the BMJ) (4).
The authors seem to have given little consideration that randomised controlled trials (RCTs) are not always the optimal vehicle for providing the best quality evidence, especially for interventions with excellent long-term outcomes where adverse outcomes can be rare and present late. As the authors are aware, the revision rate following total hip replacement is less than 5% at 10-year follow-up (5).
The potential for a placebo effect from surgery to contribute substantially to any benefit gained forms a large part of the authors’ rationale in this study. Whilst placebo effects can never be entirely ruled out, in this case, the inference is incorrect. Well-conducted RCTs (6–8) have demonstrated that the large clinical benefits experienced by patients after joint replacement persist to medium and long-term follow-up. Apart from understanding mechanisms, placebo-controlled trials are only of value when high levels of uncertainty for effect exist.
Elective orthopaedics has embraced uncertainty around treatment where it exists and an impressive portfolio of randomised trials have been completed or are ongoing (6,7,9–11). Forensic exploration of the evidence base for medical intervention is to be encouraged, but must be done responsibly, without sensationalism and with consideration of the patient as paramount.
yours sincerely
Professor John A Skinner FRCS
Vice President British Orthopaedic Association
Consultant Orthopaedic Surgeon
Stanmore
UK
Mr Tim Wilton FRCS
Medical Director, National Joint Registry of England, Wales, Northern Ireland and the Isle of Man
Consultant Orthopaedic Surgeon
Derby
UK
Professor Fares S Haddad FRCS
Editor, Bone and Joint Journal
Consultant Orthopaedic Surgeon
London
UK
Professor Hamish Simpson FRCS
Editor, Bone and Joint Research Journal
Chair, British Orthopaedic Association Research Committee
Consultant Orthopaedic Surgeon
Edinburgh
UK
Mr Robert Handley FRCS
President, British Orthopaedic Association
Consultant Trauma Surgeon
Oxford
UK
1. Evans JT, Evans JP, Walker RW, Blom AW, Whitehouse MR, Sayers A. How long does a hip replacement last? A systematic review and meta-analysis of case series and national registry reports with more than 15 years of follow-up. Lancet (London, England) [Internet]. 2019;393(10172):647–54. Available from: http://dx.doi.org/10.1016/S0140-6736(18)32531-5
2. Evans JT, Walker RW, Evans JP, Blom AW, Sayers A, Whitehouse MR. How long does a knee replacement last? A systematic review and meta-analysis of case series and national registry reports with more than 15 years of follow-up. Lancet [Internet]. 2019;393(10172):655–63. Available from: http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=med... http://oxfordsfx.hosted.exlibrisgroup.com/oxford?sid=OVID:medline&id=pmi...
3. NHS Digital. Patient Reported Outcome Measures (PROMs) [Internet]. [cited 2021 Jun 23]. Available from: https://digital.nhs.uk/data-and-information/data-tools-and-services/data...
4. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008 Sep;337:a1655.
5. National Joint Registry for England Wales Northern Ireland and the Isle of Man. NJR 17th Annual Report. 2020;(December 2019).
6. Murray DW, MacLennan GS, Breeman S, Dakin HA, Johnston L, Campbell MK, et al. A randomised controlled trial of the clinical effectiveness and cost-effectiveness of different knee prostheses: the Knee Arthroplasty Trial (KAT). Health Technol Assess [Internet]. 2014;18(19):1‐235, vii‐viii. Available from: http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=med... http://oxfordsfx.hosted.exlibrisgroup.com/oxford?sid=OVID:medline&id=pmi...
7. Beard DJ, Davies LJ, Cook JA, MacLennan G, Price A, Kent S, et al. The clinical and cost-effectiveness of total versus partial knee replacement in patients with medial compartment osteoarthritis (TOPKAT): 5-year outcomes of a randomised controlled trial. Lancet [Internet]. 2019;394(10200):746–56. Available from: http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=med... http://oxfordsfx.hosted.exlibrisgroup.com/oxford?sid=OVID:medline&id=pmi...
8. Beard DJ, Price AJ, Cook J, Fitzpatrick R, Carr A, Campbell M, et al. Total or partial knee replacement for medial osteoarthritis? Early results from the TOPKAT trial. Arthrosc - J Arthrosc Relat Surg [Internet]. 2017;33 (10 Sup:e91–2. Available from: http://ovidsp.ovid.com/ovidweb.cgi?T=JS&CSC=Y&NEWS=N&PAGE=fulltext&D=eme... http://oxfordsfx.hosted.exlibrisgroup.com/oxford?sid=OVID:embase&id=pmid...
9. Beard D, Rees J, Rombach I, Cooper C, Cook J, Merritt N, et al. The CSAW Study (Can Shoulder Arthroscopy Work?) - a placebo-controlled surgical intervention trial assessing the clinical and cost effectiveness of arthroscopic subacromial decompression for shoulder pain: Study protocol for a randomised controlled trial. Trials [Internet]. 2015 May 9 [cited 2021 Jul 31];16(1):210. Available from: https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-015-0725-y
10. Metcalfe A. A comparison of standard versus robotic total knee replacement [Internet]. [cited 2021 Jul 31]. Available from: https://www.isrctn.com/ISRCTN27624068
11. Anderson DB, Ferreira ML, Harris IA, Davis GA, Stanford R, Beard D, et al. SUcceSS, SUrgery for Spinal Stenosis: Protocol of a randomised, placebo-controlled trial. BMJ Open [Internet]. 2019 Feb 1 [cited 2021 Jul 31];9(2):24944. Available from: http://bmjopen.bmj.com/
Competing interests: All authors are practising orthopaedic surgeons and work in orthopaedic related organisations as stated. These are the British Orthopaedic Association (5500 members “Supporting surgeons Caring for patients”), National Joint Registry (over 3 million records of hip, knee, shoulder, elbow and ankle replacements) and the Editors of two of the most important trauma & orthopaedic journals. All of the authors have consulted for or received institutional research funding from orthopaedic implant manufacturers in the past. Professor Haddad states that he currently consults for Stryker and Smith & Nephew and that his Hospital receives Institutional research funds from Stryker, S&N, ZimmerBiomet & Corin. All of these are orthopaedic implant manufacturers.