John Ioannidis has written the single most cited paper in the history of PLOS Medicine(1). His research on the validity of published work in medical science has attracted world-wide acclaim. He is an internationally renowned authority on trials methodology, and is in charge of a huge programme of innovative methodological research. From our brief experiences of meeting and interacting with him he is also witty, amusing, down-to-earth, unassuming and generous in helping other workers. In short, John is a superstar in his field. But that doesn’t make him infallible; and his BMJ editorial on waste in surgical trials was, we humbly submit, wrong.
We don’t mean that it was factually wrong – of course it was entirely correct. We don’t disagree with the recommendations John makes or the vision he sets out, on the contrary we endorse these wholeheartedly. What we think was seriously mistaken was to write an entire Editorial on the very clear failings of surgical RCTs without once posing the obvious question: Why? We can both remember the last time that surgery was publicly taken to task on this issue, by the Lancet in 1996(2), and it is to say the least disappointing that, after nearly 20 years, the debate seems to have advanced so little. The underlying message of the editorial still seems to be a schoolmasterly “must try harder”. But why have surgeons apparently been messing around at the back of the classroom for so long? As a social group within medicine they are well characterised – other doctors will readily give you a view on what surgeons are like – and some characteristics it will not include are sloth, indolence, inefficiency or lack of innovation. “Must try harder” is in fact the instinctive response of most surgeons to any challenge or obstacle.
It is time it was recognised that concluding a successful RCT of a surgical treatment is substantially more difficult than conducting one of a drug or vaccine. The reasons for this have been explored and described in detail in the papers of the Balliol Collaboration(3). Surgical innovations naturally go through an early “tinkering” stage when the intervention is not yet stable, followed by an exploratory stage where learning curves and controversy over the definition and indication of the procedure are unclear. RCTs begun before these stages have been completed are inevitably at high risk of failure. The IDEAL Collaboration has consequently emphasised the need to develop an integrated evaluation pathway, encompassing at least two types of preliminary study, in order to launch surgical RCTs with a better chance of avoiding the high risk of failure reported by Chapman et al. The UK's MRC surgical Hub has done much useful work in the same area, and has also emphasised the importance of preliminary studies. Much of John’s editorial calls for improvements to the environment for clinical research which we would all endorse, but which are equally relevant to all types of trial. None of his analysis pays attention to the specific difficulties of surgical studies, yet this is surely where the explanations lie for the comparatively poor performance of surgical trials reported in Chapman’s paper. In order to avoid the fate of Bill Murray’s character in “Groundhog day”* we urge that the surgical community (and those who fund and publish its research) to do more to nurture, support and develop the preliminary studies which are needed in order for surgical RCTs to do better.
1. Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2: e124. doi:10.1371/journal.pmed.0020124.
2.Horton R. Surgical research or comic opera: questions, but few answers. Lancet. 1996 Apr 13;347(9007):984-5.
3. McCulloch, P., Altman DG, Campbell WB, Flum DR, Glasziou P, Marshall JC, Nicholl J., No surgical innovation without evaluation: the IDEAL recommendations. Lancet, 2009. 374(9695): p. 1105-12.
Competing interests:
Peter McCulloch is Chair of the IDEAL Collaboration, which works to improve methodology in trials of surgery and interventional therapies.
19 January 2015
Peter G McCulloch
Academic surgeon
Richard Lehmann
University of Oxford
Nuffield Department of Surgical Science, Level6, John Radcliffe Hospital, Oxford OX3 9DU
Rapid Response:
John Ioannidis has written the single most cited paper in the history of PLOS Medicine(1). His research on the validity of published work in medical science has attracted world-wide acclaim. He is an internationally renowned authority on trials methodology, and is in charge of a huge programme of innovative methodological research. From our brief experiences of meeting and interacting with him he is also witty, amusing, down-to-earth, unassuming and generous in helping other workers. In short, John is a superstar in his field. But that doesn’t make him infallible; and his BMJ editorial on waste in surgical trials was, we humbly submit, wrong.
We don’t mean that it was factually wrong – of course it was entirely correct. We don’t disagree with the recommendations John makes or the vision he sets out, on the contrary we endorse these wholeheartedly. What we think was seriously mistaken was to write an entire Editorial on the very clear failings of surgical RCTs without once posing the obvious question: Why? We can both remember the last time that surgery was publicly taken to task on this issue, by the Lancet in 1996(2), and it is to say the least disappointing that, after nearly 20 years, the debate seems to have advanced so little. The underlying message of the editorial still seems to be a schoolmasterly “must try harder”. But why have surgeons apparently been messing around at the back of the classroom for so long? As a social group within medicine they are well characterised – other doctors will readily give you a view on what surgeons are like – and some characteristics it will not include are sloth, indolence, inefficiency or lack of innovation. “Must try harder” is in fact the instinctive response of most surgeons to any challenge or obstacle.
It is time it was recognised that concluding a successful RCT of a surgical treatment is substantially more difficult than conducting one of a drug or vaccine. The reasons for this have been explored and described in detail in the papers of the Balliol Collaboration(3). Surgical innovations naturally go through an early “tinkering” stage when the intervention is not yet stable, followed by an exploratory stage where learning curves and controversy over the definition and indication of the procedure are unclear. RCTs begun before these stages have been completed are inevitably at high risk of failure. The IDEAL Collaboration has consequently emphasised the need to develop an integrated evaluation pathway, encompassing at least two types of preliminary study, in order to launch surgical RCTs with a better chance of avoiding the high risk of failure reported by Chapman et al. The UK's MRC surgical Hub has done much useful work in the same area, and has also emphasised the importance of preliminary studies. Much of John’s editorial calls for improvements to the environment for clinical research which we would all endorse, but which are equally relevant to all types of trial. None of his analysis pays attention to the specific difficulties of surgical studies, yet this is surely where the explanations lie for the comparatively poor performance of surgical trials reported in Chapman’s paper. In order to avoid the fate of Bill Murray’s character in “Groundhog day”* we urge that the surgical community (and those who fund and publish its research) to do more to nurture, support and develop the preliminary studies which are needed in order for surgical RCTs to do better.
1. Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2: e124. doi:10.1371/journal.pmed.0020124.
2.Horton R. Surgical research or comic opera: questions, but few answers. Lancet. 1996 Apr 13;347(9007):984-5.
3. McCulloch, P., Altman DG, Campbell WB, Flum DR, Glasziou P, Marshall JC, Nicholl J., No surgical innovation without evaluation: the IDEAL recommendations. Lancet, 2009. 374(9695): p. 1105-12.
Competing interests: Peter McCulloch is Chair of the IDEAL Collaboration, which works to improve methodology in trials of surgery and interventional therapies.