FDA official: “clinical trial system is broken”BMJ 2013; 347 doi: https://doi.org/10.1136/bmj.f6980 (Published 05 December 2013) Cite this as: BMJ 2013;347:f6980
All rapid responses
I take exception to the assertions raised in interview by Marciniak, BMJ- 5 December 2013. In his three topics I believe he conflates design and analysis issues with his complaints about lack of FDA vigilance. I too was an FDA reviewer - for over 15 years, and found the opposite, i.e., appropriate attention to these issues.
1-“Seeds of disillusion”: Adding a secondary endpoint after the trial had begun.
There is a literature (on adaptive designs) on what is and what is not acceptable regarding interim protocol changes, and most would agree that adding a secondary endpoint in this manner is not acceptable. These protocol provisos and their timing are, of course, critically important, and transparency is crucial to enabling a convincing interpretability of results. Currently, this area of trial design and reporting remains very deficient. For airtight assurance of full disclosure of all protocol provisions, amendments, and statistical analysis plan, one needs a time-sensitive, formal repository of all these materials which, upon completion of the trial, becomes publicly available (1). As yet, there is no such repository.
2-“Missing data crusade”: The impact of missing data
Missing data is the Achilles heel of trial design, and there is a large literature on missing data but – except for winning by worst case scenario – there is no satisfactory statistical solution if the missing data are informative, which they usually are. Modeling is often proposed, but it carries with it assumptions that cannot themselves be verified. A practical approach, albeit still with a degree of discretion, is to impose increasingly severe assumptions about how different the treatment effect is in the missing data cohort versus the non-missing cohort, and thus see how robust the conclusion is. The best solution, of course, is avoidance by design. For example, use of a binary – success/failure – endpoint with a protocol proviso that missing data patients are deemed failures, circumvents the problem.
A win by “worst case scenario” is when the conclusion still holds when missing control patients are given the best outcome and test missing drug patients are given the worst outcome. For example, the first large hypertension trial (2) found a significant difference in all morbid events, 2/73 in the test arm versus 27/70 in the control arm (p<0.001< Fisher Exact). Patient with missing data were 5/73, test, versus 7/70, control. Adding these as events if in test arm and as non-events if in control yields a 7/73 versus 27/70 worst case analysis comparison, still significant at P<0.001.
In my field, arthritis, we were constantly plagued with issues surrounding missing data. Was the positive conclusion of the trial due to the drug or due to the missing data pattern? The evidence to support a structural retardation claim, a slowing of x-ray progression, is a case in point. The first rheumatoid arthritis drug to achieve this claim was leflunomide, and its x-ray trials had substantial missing data. I embarked on a laborious process of designing an imputation analysis as a sensitivity analysis to address how deviant the drug effect in the missing data patients could be and still have the overall drug effect conclusion remain statistically signficant (3). In the end this analysis demonstrated that the conclusion would still hold even if the x-ray effect in the missing patients was in the opposite direction as that seen in the patients without missing data. This sensitivity analysis seemed clinically reassuring and the claim was granted in the product label.
The angiotensin II receptor antagonists meta-analysis, like many meta-analyses, presents problems, both clinical and statistical, including small effect sizes with wide confidence intervals, problematic for observational studies. The same can be said for the meta-analyses of rosiglitazone, whose evidence base has been augmented with the outcome study, RECORD. This was an open label, non-inferiority trial with asymmetrical insulin use across arms and a primary endpoint that included cardiovascular hospitalizations – all design aspects that would, and did, lead to threats to interpretability. Diabetes is particularly difficult to study drugs in randomized settings because there is an additional clinical mandate to target glucose control and if such targeting is differentially applied, the conclusion will be confounded. Finally, there is yet to emerge – despite more than three decades of use – a consensus on the level of evidence desired of meta-analyses of efficacy (4), let alone that for meta-analyses with safety (5).
1.Lassere M, Johnson K: The power of the protocol. Lancet 2002;360:1620-22.
2.Veterans Administration Cooperative Study Group on antihypertensive agents: Effects of treatment on morbidity in hypertension. I. Results in patients with diastolic blood pressures averaging 115 through 129 mm Hg. JAMA 1967,202:166-122.
4.Peto R. Discussion: Why do we need systematic overviews of randomized Trials? Statist Med 1987; 6:241-244.
5.Memorandum from Robert Temple to Janet Woodcock re. Data on Rosiglitazone, pp 428-450, Pre-meeting material for 5-6 June 2013 Endocrine and Metabolic Drugs Advisory Committee. http://www.fda.gov/downloads/AdvisoryCommittees/CommitteesMeetingMateria...
Competing interests: No competing interests