Intended for healthcare professionals

  1. Research
  2. Design characteristics...
  3. Design characteristics, risk of bias, and reporting of randomised controlled trials supporting approvals of cancer drugs by European Medicines Agency, 2014-16: cross sectional analysis
CCBYNC Open access
Research

Design characteristics, risk of bias, and reporting of randomised controlled trials supporting approvals of cancer drugs by European Medicines Agency, 2014-16: cross sectional analysis

BMJ 2019; 366 doi: https://doi.org/10.1136/bmj.l5221 (Published 18 September 2019) Cite this as: BMJ 2019;366:l5221

Linked Editorial

Flawed evidence underpins approval of new cancer drugs

Linked Opinion

Gauging the validity of cancer drug trials: a call for collaboration
  1. Huseyin Naci, assistant professor of health policy and Harkness fellow1 2,
  2. Courtney Davis, reader3,
  3. Jelena Savović, senior research fellow in evidence synthesis4 5,
  4. Julian P T Higgins, professor of evidence synthesis4 5 6,
  5. Jonathan A C Sterne, professor of medical statistics and epidemiology4 6,
  6. Bishal Gyawali, fellow and assistant professor of public health sciences2 7,
  7. Xochitl Romo-Sandoval, research assistant1,
  8. Nicola Handley, research assistant3,
  9. Christopher M Booth, professor of oncology7
  1. 1Department of Health Policy, London School of Economics and Political Science, London WC2A 2AE, UK
  2. 2Program on Regulation, Therapeutics, and Law (PORTAL), Division of Pharmacoepidemiology and Pharmacoeconomics, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, USA
  3. 3Department of Global Health and Social Medicine, King’s College London, London, UK
  4. 4Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
  5. 5The National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West) at University Hospitals Bristol NHS Foundation Trust, Bristol, UK
  6. 6National Institute for Health Research Bristol Biomedical Research Centre, Bristol, UK
  7. 7Cancer Research Institute, Queen’s University at Kingston, Kingston, Ontario, Canada
  1. Correspondence to: H Naci h.naci{at}lse.ac.uk (or @huseyinnaci2 on Twitter)
  • Accepted 17 July 2019

Abstract

Objective To examine the design characteristics, risk of bias, and reporting adequacy of pivotal randomised controlled trials of cancer drugs approved by the European Medicines Agency (EMA).

Design Cross sectional analysis.

Setting European regulatory documents, clinical trial registry records, protocols, journal publications, and supplementary appendices.

Eligibility criteria Pivotal randomised controlled trials of new cancer drugs approved by the EMA between 2014 and 2016.

Main outcome measures Study design characteristics (randomisation, comparators, and endpoints); risk of bias using the revised Cochrane tool (bias arising from the randomisation process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported result); and reporting adequacy (completeness and consistency of information in trial protocols, publications, supplementary appendices, clinical trial registry records, and regulatory documents).

Results Between 2014 and 2016, the EMA approved 32 new cancer drugs on the basis of 54 pivotal studies. Of these, 41 (76%) were randomised controlled trials and 13 (24%) were either non-randomised studies or single arm studies. 39/41 randomised controlled trials had available publications and were included in our study. Only 10 randomised controlled trials (26%) measured overall survival as either a primary or coprimary endpoint, with the remaining trials evaluating surrogate measures such as progression free survival and response rates. Overall, 19 randomised controlled trials (49%) were judged to be at high risk of bias for their primary outcome. Concerns about missing outcome data (n=10) and measurement of the outcome (n=7) were the most common domains leading to high risk of bias judgments. Fewer randomised controlled trials that evaluated overall survival as the primary endpoint were at high risk of bias than those that evaluated surrogate efficacy endpoints (2/10 (20%) v 16/29 (55%), respectively). When information available in regulatory documents and the scientific literature was considered separately, overall risk of bias judgments differed for eight randomised controlled trials (21%), which reflects reporting inadequacies in both sources of information. Regulators identified additional deficits beyond the domains captured in risk of bias assessments for 10 drugs (31%). These deficits included magnitude of clinical benefit, inappropriate comparators, and non-preferred study endpoints, which were not disclosed as limitations in scientific publications.

Conclusions Most pivotal studies forming the basis of EMA approval of new cancer drugs between 2014 and 2016 were randomised controlled trials. However, almost half of these were judged to be at high risk of bias based on their design, conduct, or analysis, some of which might be unavoidable because of the complexity of cancer trials. Regulatory documents and the scientific literature had gaps in their reporting. Journal publications did not acknowledge the key limitations of the available evidence identified in regulatory documents.

Footnotes

  • Contributors: HN conceived and designed the study. XRS, NH, and HN collected data and completed the risk of bias assessments. HN independently verified all collected data including risk of bias assessments, undertook the primary analysis, and drafted the manuscript. All authors contributed to subsequent iterations. All authors provided critical input on the manuscript and approved the final version for publication. XRS and NH contributed equally. HN is guarantor. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

  • Funding: HN is 2018-19 UK Harkness Fellow in Health Care Policy and Practice, funded by the Commonwealth Fund. HN and CD are supported by the Higher Education Funding Council in England. XRS and NH were funded in part by Health Action International. JPTH and JACS are National Institute for Health Research (NIHR) senior investigators, are supported by NIHR Biomedical Research Centre at University Hospitals Bristol NHS Foundation Trust and the University of Bristol, and are members of the Medical Research Council (MRC) Integrative Epidemiology Unit at the University of Bristol. JS and JPTH were supported by the NIHR Collaboration for Leadership in Applied Health Research and Care West (CLAHRC West) at University Hospitals Bristol NHS Foundation Trust. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the UK Department of Health and Social Care. BG’s fellowship at the Program on Regulation, Therapeutics, and Law (PORTAL) was funded by Arnold Ventures. CMB is supported as the Canada Research Chair in Population Cancer Care. Funders did not have any role in the study design; the collection, analysis, and interpretation of data; the writing of the report; or the decision to submit the article for publication. All authors had full access to all of the data in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf and declare: support from the Commonwealth Fund, Higher Education Funding Council in England, Health Action International (HAI), National Institute for Health Research (NIHR) Biomedical Research Centre at University Hospitals Bristol NHS Foundation Trust and the University of Bristol, NIHR Collaboration for Leadership in Applied Health Research and Care West (CLAHRC West) at University Hospitals Bristol NHS Foundation Trust, and Arnold Ventures for the submitted work; CD reports being a member of the non-profit organisation HAI, and occasionally attends meetings of the European Medicines Agency’s Patients and Consumers Working Party Group as HAI’s alternative representative; XRS reports grants from HAI during the conduct of the study and personal fees from Sanofi Aventis, outside the submitted work; NH reports grants from HAI during the conduct of the study; the other authors declare no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years, no other relationships or activities that could appear to have influenced the submitted work.

  • Ethical approval: Not required.

  • Data sharing: No additional data are available.

  • The lead author (HN) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

View Full Text
Back to top