Problems of stopping trials earlyBMJ 2012; 344 doi: https://doi.org/10.1136/bmj.e3863 (Published 15 June 2012) Cite this as: BMJ 2012;344:e3863
All rapid responses
Guyatt et al report on a potential problem with overestimate of treatment effect due to termination of a clinical trial at an interim. Guyatt et al note that statisticians have described this well known phenomenon in the past, citing the paper by Pocock Hughes. Particularly surprising is that Guyatt et al do not report that statistical corrections for the bias in the treatment over-estimates are available for most, if not all, formal interim statistical analysis procedures, for example, for O'brien Fleming. One citation (there are many more) to a bias correct is Qu, Roger P., and David L. DeMets. "Bias correction in group sequential analysis with correlated data." Statistica Sinica (1999): 939-952.
Competing interests: No competing interests
Generally speaking, research strives to fill the gap between ignorance and knowledge. In particular, clinical research struggles to provide solid and reliable evidences for the best management of real patients. Moreover, the pathway between ignorance, mere suppositions, explorative hypotheses and trustworthy evidences is not straightforward, but it is mined by confounding factors such as the intrinsic complexity of human diseases and the stochastic variability of everyday contingency. In this sense, formal transparency and methodological integrity in the study protocol design, data analysis and results reporting are crucial for the safe translation of research evidences in clinical practice. In the current system of knowledge delivery, warranty of evidences’ good quality should be provided by the pre-publication peer-review process to whom research papers are submitted by journals’ editors and specific fields’ experts; moreover the post-publication correspondence represents a valuable opportunity for the discussion and validation of evidences among the scientific community. Anyway, research papers are often weakened by methodological mistakes , and strict editorial instructions seem to discourage open and widespread post-publication correspondence .
We recently read with interest the results of the FAME-II Trial, a randomized controlled trial (RCT) supported by St. Jude Medical®, in which fractional flow reserve (FFR) – guided percutaneous coronary intervention (PCI) was compared to the best available medical therapy in the treatment of stable coronary artery disease (CAD) . The authors reported that the recruitment was halted prematurely because of a significantly higher frequency of the primary end-point (a composite of death from any cause, nonfatal myocardial infarction, or unplanned hospitalization leading to urgent revascularization during the first 2 years) in the medical therapy group. That result was driven mostly by urgent revascularizations (49/441 vs 7/447 in the PCI group; 95% CI 0.13[0.06-0.30]; p<0.001). These striking data received great resonance and were published online with priority on the New England Journal of Medicine (NEJM) website to coincide with data presentation at the European Society of Cardiology annual meeting.
However, something did not convince us. It is indeed demonstrated that RCTs stopped early for benefit (truncated RCTs) show larger treatment effects than non-truncated RCTs, with greater overestimations associated with total number of events inferior to 200 [4,5]. It is also well known that truncated RCTs sometimes led to fallacious results . Moreover, the CONSORT 2010 Statement recommends reporting the methods employed in the decision of early interruption of a RCT . Our main concern was that the FAME-II study protocol did not schedule any interim analysis and that no formal stopping rule was pre-specified for the trial suspension, leaving the whole responsibility of such a decision to the Data Safety and Monitoring Board (DSMB) on the basis of data provided by a third party Clinical Research Organization (CRO). The attention to such a detail is limited to the discussion of the results, where it is briefly stated that considering that “the difference in the primary outcome between the two treatment groups was large and was steadily increasing over time … the DSMB believed that exposing more patients with functionally significant stenoses to the risk of urgent revascularization was inappropriate”. However, given the small number of end-point-leading events (56) and the absence of a detailed description of interruption criteria, one could then argue that the FAME-II may belong to that category of above-mentioned misleading RCTs. For these reasons and in consideration of the potential repercussion of this study on the treatment of stable CAD we decided to share our doubts with the scientific community writing a letter to NEJM editors, kindly exhorting the FAME-II authors to provide methods and salient data that supported the early termination of the RCT. We believed, and still we do, that in this way the results of the FAME-II Trial would gain formal power and could transparently become a solid landmark in the management of this highly prevalent disease. Unfortunately, the letter was declined due to lack of space available for correspondence. Moreover, none of the subsequently published letters mention this issue [8,9]. This is fairly unpleasant, as it seems that critical discussion about methodological integrity of clinical research does not deserve room in a high-impact medical journal such as NEJM.
It is distinctly becoming clear that fully reliable evidences are crucial for an appropriate medical practice. Recently the British Medical Journal editor in chief announced the decision to publish from January 2013 only clinical trials where there is a commitment to make the relevant anonymised patient data available on reasonable request . Perhaps a similar critical spirit should animate the post-publication peer-review process . In this sense we insist in encouraging the FAME-II authors to answer the raised questions, hoping to find some room in your journal.
1. Altman D. Poor-quality medical research. JAMA. 2002;287:1142–1143.
2. Altman DG. Unjustified restrictions on letters to the editor. PLoS Med. 2005;2:e126.
3. De Bruyne B, Pijls NHJ, Kalesan B, et al. Fractional Flow Reserve–Guided PCI versus Medical Therapy in Stable Coronary Disease. N Eng J Med. 2012;367:911–1001.
4. Montori V, Devereaux P. Randomized trials stopped early for benefit: a systematic review. JAMA. 2005;294:2203–2209.
5. Bassler D, Briel M, Montori V. Stopping randomized trials early for benefit and estimation of treatment effects. JAMA. 2010;303:1180–7.
6. Guyatt GH, Briel M, Glasziou P, Bassler D, Montori VM. Problems of stopping trials early. BMJ. 2012;344:e3863.
7. Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c869.
8. Goodney PP, Woloshin S, Schwartz LM. Fractional Flow Reserve–Guided PCI in Stable Coronary Disease. N Eng J Med. 2012;367:2355–2356.
9. Helft G, Choussat R, Zaman A. Fractional flow reserve-guided PCI in stable coronary disease. N Eng J Med. 2012;367:2355–6.
10. Godlee F. Clinical trial data for all drugs in current use. BMJ. 2012;345:e7304.
Competing interests: No competing interests
This paper raises the important philosophical distinction between evidence and truth. The point here seems to be that statistical means of discerning the truth about the systems in point cannot be relied upon to access the underlying truth until there are sufficient events for a true picture of the underlying reality to be formed. Before this sufficient accretion of data there is a great deal of noise in the signal.
The moral problem is who, within the trial community, is obliged to guard against this type of misperception impacting adversely upon the scientific evidence base and clinical care?
I would argue that it should not fall to those who might face a conflict of interests in reaching such a decision. Thus researchers, research funding bodies and journals are perhaps not best placed to attend to this problem. The authors rightly criticise guideline committees for seeking to offer advice when the underlying evidence base, whilst in existence, is insufficient to permit the underlying truth to be accurately discerned. (1)
One option might be to require a more powerful role for research ethics committees in scrutinising the scientific validity of the trial stopping rules on the basis that this is actually an ethical issue arising from the scientific design.
(1) See Mohindra RK. A case of insufficient evidence equipoise: the NICE guidance on antibiotic prophylaxis for the prevention of infective endocarditis. J Med Ethics 2010;36:567-570 doi:10.1136/jme.2010.036848
Competing interests: I am an expert member of a local research ethics committee