How to use FDA drug approval documents for evidence syntheses
BMJ 2018; 362 doi: https://doi.org/10.1136/bmj.k2815 (Published 10 July 2018) Cite this as: BMJ 2018;362:k2815
- Aviv Ladanie, research fellow1 2,
- Hannah Ewald, research fellow1 2 3,
- Benjamin Kasenda, senior scientist1 4,
- Lars G Hemkens, senior scientist1
- 1Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel and University of Basel, Basel, Switzerland
- 2Swiss Tropical and Public Health Institute (Swiss TPH), Basel, Switzerland
- 3University Medical Library, University of Basel, Basel, Switzerland
- 4Departments of Medical Oncology and of Haematology, University Hospital Basel and University of Basel, Basel, Switzerland
- Correspondence to: L G Hemkens lars.hemkens{at}usb.ch
- Accepted 18 May 2018
Summary box
There is compelling evidence that published trial information is selectively reported and that results not showing favourable effects of the tested treatments often remain unpublished
Clinical trial information published by regulatory authorities such as the US Food and Drug Administration (FDA) may help to reduce such reporting biases
FDA approval documents are long and do not follow the typical structure of medical journal articles, which may discourage reviewers from using them for evidence syntheses
Our practical guidance on how to efficiently identify and use approval documents to find the relevant information may help promoting the use of this valuable data source for evidence syntheses
Publicly accessible approval documents published by the US Food and Drug Administration (FDA) provide important insights into reporting biases in articles published in peer reviewed medical journals,1234 but the FDA data can also be used to directly minimise the impact of such biases on the results and conclusions in evidence syntheses,5 obtain information not disclosed in published clinical trials reports,6 and identify unpublished clinical trials to increase precision of effect estimates.7
For example, almost 20 years ago, Man-Son-Hing and colleagues5 showed that incorporating unpublished trials into a meta-analysis on quinine for nocturnal leg cramps substantially reduced the estimated efficacy. The bias occurred because almost all published trials had larger effects than the unpublished studies. Similarly, Turner and colleagues found that 22 (31%) of 71 trials discussed in FDA approval documents of 12 antidepressants were not published, and that publication was closely associated with results favouring the experimental drugs.1 Hart and colleagues showed that updating meta-analyses with unpublished trial data from FDA approval documents changed drug efficacy estimates in 38 of 41 cases (93%) towards both lower and higher efficacy.4 Rising and colleagues not only revealed that approval trials are often unpublished, in particular when they suggest unfavourable outcomes for the experimental intervention, but also that the published information is incomplete because results were omitted from the papers.2 MacLean and colleagues, on the other hand, demonstrated that incorporating unpublished trial data in meta-analyses does not necessarily change treatment effect estimates.7 McDonagh and colleagues show various examples where information from FDA documents would alter conclusions of drug effectiveness reviews, not only by identifying unpublished studies, but also by providing unpublished information on benefits and harms found in published trials and by reporting results of independent analyses conducted by the FDA.6
Overall, such examples illustrate that unpublished FDA trial data have the potential to change the results of evidence syntheses and can provide useful information that would otherwise be unavailable.68 It may help to better understand the strengths and weaknesses of a trial that only a regulator could discover, given the regulator’s access to the original study data and the original trial protocols (drug developers must submit the trial protocols before they initiate the approval trials9).
However, regulatory data are rarely used in evidence syntheses. A survey estimated that only 24 of 794 Cochrane reviewers (3%) who had searched for unpublished clinical trials had gathered information from health authorities.10 The survey authors hypothesise that “some authors might not be aware of the amount of accessible data at regulatory agencies.” Others emphasise that FDA approval documents are difficult to access and navigate.11121314
Attempts to promote the adoption of regulatory data as a viable source of trial information have been made recently, including a guide on where and how to retrieve FDA approval documents,13 a description of the content in FDA and European Medicines Agency (EMA) documents,15 and the dissemination of FDA approval documents in a more accessible format via the recently launched OpenTrialsFDA platform.16
The sheer amount of information encountered in an FDA approval document, usually hundreds of pages not following the typical structure of medical journal articles, may indeed confuse and discourage reviewers from using them for evidence syntheses.
Guidance could be provided by the publisher of these documents (the FDA), by leading organisations advocating for systematic reviews of healthcare interventions (particularly the Cochrane Collaboration), or by reviewers with experience in using FDA approval documents. To our knowledge, the FDA has not published any description of the content and structure of their approval documents, in contrast to the EMA and the Australian Therapeutic Goods Administration (TGA).17 Neither does the Cochrane Handbook provide advice in this direction, and—according to the changelog18—the updated version (5.2, June 2017) will not be addressing this issue either.19
We have used FDA approval documents in meta-epidemiological projects, including an ongoing analysis of 92 anticancer agents approved by the FDA between 2000 and 2016. Here, we share our knowledge and describe how we navigate such documents efficiently. We also indicate where one can expect to find information typically relevant for evidence syntheses.
The target users of this guide are mainly authors of evidence syntheses who intend to collect and synthesise evidence on a given topic in a systematic and transparent manner, such as in systematic reviews or meta-analyses. This guidance can assist in identifying potentially unpublished drug trials, obtaining additional information that is unavailable from published clinical trial reports, or for cross-checking information reported in journal articles. However, it does not cover the subsequent indispensable steps of evidence synthesis including the quality assessment, for which further detailed guidance would be needed.
Provenance of this guidance
We describe the structure and content of FDA approval documents based on our experience from various meta-research projects, including the ongoing CEIT-cancer project in which we evaluate the evidence base from pre-marketing clinical trials of 92 anticancer agents. In this project, we systematically acquire FDA approval packages, peruse approval documents, and extract trial characteristics as well as treatment effect estimates. In addition, we pilot tested the applicability of our guidance for several drugs approved for neurological, cardiovascular, psychiatric, endocrine, and rheumatological disorders and can confirm its validity across medical specialties.
The FDA drug approval package
To obtain marketing authorisation for newly developed drugs and therapeutic biologic products (herein referred to as “drugs”), companies have to submit a New Drug Application (NDA) or Biologics License Application (BLA) to the FDA and provide information about the drug’s quality, safety, and efficacy.20 The FDA reviews this information and—for drugs that are ultimately approved for marketing—publishes these reviews (albeit in a form redacted of some information) online in the drugs@FDA database as PDF documents.21 Documents pertaining to a single approved product are organised in “approval packages.” There are guidance22 and policy documents23 which provide a deeper understanding of the FDA processes and evolving procedures. The review process is addressed by a “Good Review Practice” document within the FDA’s Center for Drug Evaluation and Research (CDER) “that discusses any aspect related to the process, format, content and/or management of a product review.”24
Approval packages are available for prescription and over-the-counter drugs as well as “drug-like” agents (such as therapeutic biologics, which include antibodies, cytokines, growth factors, enzymes, and immunomodulators). For a detailed list of what is included in the drugs@FDA database, look in the “frequently asked questions.”25 Approval documents are only sporadically available for drugs approved before 1997, and are rarely available for supplemental indications (that is, indications approved after a drug has received its initial marketing authorisation). Approval documents for devices and non-therapeutic biologics (such as vaccines, blood, and blood products) are not regulated by the CDER and are not addressed by this guidance.
Finding and accessing approval packages
We provide guidance on determining whether FDA approval packages are available for a given drug of interest and how to find the corresponding FDA approval package in the supplemental material on bmj.com (“Part 1: How to access FDA approval packages”).
Document types in approval packages
We are aware of about 20 different document types included in FDA approval packages (box 1). The medical review is typically most relevant for evidence syntheses. The medical reviews (sometimes referred to as clinical reviews) usually contain sufficient information for identifying and selecting pertinent trials as well as main information about clinical trial characteristics, statistical analyses and results. However, some important details may be revealed only after thorough perusal of the entire approval package. Since 2016, the medical review document of recently approved drugs is often merged with other document types in a single “multi-discipline review” document. Its content can now be found under the table of contents heading “Statistical and clinical evaluation.” If the medical review document is missing, incomplete, or illegible, or when more in-depth analysis is required, the other documents available in the FDA approval package may provide further information (see below).
Overview of documents that may be included in FDA approval packages:
Administrative documents
Approval letter
Botanical review
Chemistry, manufacturing, and controls review / Chemistry review
Clinical pharmacology and biopharmaceutics review / Clinical pharmacology review / Pharmacology review
Cross discipline team leader review
Drug label*
Environmental assessment review
Immunogenicity review
Medical review / Clinical review*
Multi-discipline review*
Medication guide / Patient package insert
Microbiology review
Name review
Office director memo
Officer/employee list
Other review
Risk assessment and risk mitigation review
Statistical review*
Summary review
Toxicology review
* Usually most relevant documents for evidence syntheses
Medical review structure
The medical review document structure has evolved over the years, but we identified three general document structures (fig 1) used since approximately 2004. For older drugs and biologicals, there is no such consistent document structure.
Overview of FDA medical review document structures, with the sections most relevant for evidence syntheses highlighted
The actual document structure may deviate slightly from these general structures, such as by section heading (thus, the section “Review of efficacy” may be titled “Efficacy evaluation” or “Integrated review of efficacy”); grouping of information (for example, section “Clinical pharmacology” may be a subsection of “Significant issues from other review disciplines” or it may exist as a standalone section); or sequence of sections.
We generally suggest that reviewers first try to locate the table of contents (some older reviews may have none) and make themselves familiar with the document structure. Sometimes there may be two tables of contents in a medical review document. This may indicate that the original application for marketing authorisation was declined by the FDA and that the agency re-evaluated the drug in a second review. Triptorelin (Trelstar) is such an example (with tables of contents on pages 2 and 41).26 In this situation, both medical reviews should be scrutinised because the data may differ between the two. Whether to use the superseded or the updated version can only be decided in the context of the research question. An explanation why triptorelin was approved only after a second review cycle can be found in the regulatory history, usually described in sections titled “Introduction,” “Background,” or similar (fig 1).
Data collection
Where to find relevant clinical trials and their characteristics
Trials submitted to the FDA to support approval are presented and discussed in detail in the purple highlighted sections in figure 1. There is typically a tabular overview with brief information on individual trials such as the target population, interventions, comparators, outcomes, time frame, setting, and study design.
Many trials have multiple trial names or identifiers (for example, C0743T09 and PHOENIX 2 in the case of ustekinumab (Stelara) (see medical review page 17)27 and CLEOPATRA, TOC4129g, or WO20698 in the case of pertuzumab (Perjeta) (see medical review page 30)28). Knowing all trial identifiers may facilitate locating the corresponding record on clinicaltrials.gov or identifying reports of trials published in journals. For example, the pivotal trial of ustekinumab can be found as “PHOENIX 2” on PubMed but only with detours on clinicaltrials.gov, whereas it can be quickly found with its identifier C0743T09 on clinicaltrials.gov but not in PubMed.
The purple highlighted sections are comparable with the methods section in medical journals in terms of content, with information on trial design (objectives, geographical distribution of sites), trial population (eligibility criteria), interventions, trial endpoints (definitions, outcome assessment), and statistical methods (sample size, calculation of effect estimates, and details on interim, subgroup, and sensitivity analyses). Excerpts from trial protocols and a history of protocol amendments may also be found in these sections. They allow, for example, assessing pre-specification of endpoints or subgroups. For drugs approved before 2007, additional information about trial methods (and results) may be available in the appendix (fig 1).
Where to find trial results
Results for efficacy endpoints are reported in the pink highlighted sections in figure 1. The sections start with details of the trial population and enrolment, often including a flow diagram, site-specific enrolment information, and summary statistics of patient characteristics.
Not all trials presented in the purple highlighted sections (fig 1) have results, either because they are of no relevance for drug approval, they are ongoing, or they address a different indication. Such trials may still have been used to evaluate the drug’s safety profile and to better understand the risk for adverse events. The methods and results of the safety review are presented in the green highlighted sections (fig 1).
We provide a step-by-step instruction (including a working example) on where to find relevant clinical trials and their characteristics in the supplemental material on bmj.com.
Extraction of trial results
Results for various endpoints from several pre-specified and exploratory analyses are presented in the results section of the medical review. The decision about which one to choose should be made depending on the research aims. We present some general examples of the various types of analyses reported in medical reviews to facilitate pre-specification of the extractions and analyses of interest (box 2). We provide a step-by-step instruction (including a working example) on where to find trials results in the supplemental material.
Diversity of outcome analyses commonly seen in FDA medical reviews
Intention-to-treat analyses versus per protocol or other analyses
Different data cut-off-dates (for example, interim analyses, final analyses, follow-up analyses)
All study sites versus subsets of sites (for example, geographic region)
Local versus central outcome assessments
Analyses conducted by the FDA versus analyses conducted by sponsors
Analyses adjusted for covariates versus unadjusted analyses
Pre-specified analyses versus post-hoc analyses
Further and more detailed information
Additional information on statistical analyses and sample size calculation are often provided in the statistical review document which is also included in FDA approval packages. FDA guidance states: “applicants are expected to submit data of high quality and make it possible for the FDA to reproduce their results. In turn, FDA reviewers should provide adequate documentation so that the applicant or another data user could reproduce their independent findings.”29 The statistical review often includes details of such re-analyses, for example: “whether it is possible to reproduce the primary analysis dataset, and in particular the primary endpoint, from the original data source.”29
Sometimes there are comments by the FDA medical or statistical reviewer which may be informative for the assessment of the quality of evidence—for example, to address potential risks of bias or to discuss the adequacy of comparator interventions.
Recently (March 2018), the FDA announced to publish more detailed information, such as Clinical Study Reports (CSR). This process is at an early stage, and we hope that such promising evidence will be available for future drug approvals, but also from drugs approved in the past. There is currently the option to obtain such CSRs through Freedom of Information Act requests.30
Challenges and potential solutions
The reporting quality in FDA approval documents has generally improved over time, but inconsistencies and contradictions across medical review sections and document types occur. Independent data extractions by two reviewers may be helpful to overcome this problem and to increase the reliability of the data. Recording on which document pages the extracted information was found may facilitate the consensus steps. A general problem of using the approval package is missing, inconsistent, or selectively reported information. A specific problem is that some of the information in FDA reports may be redacted for various reasons.15
Schroll and colleagues reported that, in their systematic sample of drugs approved between 2011 and 2012, “crucial information about safety concerns and nonapproved indications were redacted in the FDA reports.”15 Therefore, utmost attention and careful evaluation on a case by case basis is required to assess potential biases resulting from incomplete information. In this regard, a close evaluation of the concerns of the various FDA reviewers in assessing efficacy and safety may provide valuable information. Drugs approved before 2007 are more prone to suboptimal reporting. Options to deal with missing information for outcome data include indirect calculation of effect estimates of time-to-event endpoints (for example arm-specific number of events, point estimates, P values, Kaplan-Meier curves),3132 and juxtaposition of the FDA approval document information with corresponding trial reports in published journal articles, trial registries (such as clinicaltrials.gov), or material from other approval agencies.
This guidance aims to reduce barriers to use this evidence by making access and navigation easier. However, important questions remain about how to best integrate this valuable source of information into evidence syntheses. We touch only briefly on crucial subsequent steps of evidence synthesis, for which further detailed guidance is needed. In particular, thorough assessment of the quality and potential biases of the information provided in approval documents is indispensable. Overall, we believe that, despite such limitations, the consideration of approval documents can strengthen evidence syntheses of drug interventions.
Recommendations and some major points to consider
First, find out whether relevant FDA approval packages are available and how to access them
When older documents are not searchable, consider using text recognition software
The medical review is a key document and good starting point
Try to locate the table of contents of an FDA approval document and get familiar with the overall document structure
Identify pertinent trials and main information on trial characteristics and results in the medical review
Note all trial names and identifiers, where possible
Independent data extraction by multiple reviewers and noting the document pages where information was found may be helpful
Consider further approval documents such as the statistical review, which may reveal important details
Re-analyses by the FDA and comments of the different FDA reviewers may provide valuable insights for quality of evidence assessment.
FDA reviewers have a unique view on the original research data that only a regulator can have
Redacted information is common and requires special attention
Compare data extracted from approval information with data from other sources
Always assess the quality of evidence carefully, which sometimes requires scrutiny of the entire approval package to clarify risks of bias
Acknowledgments
Parts of this article were presented at the Evidence Live conference in Oxford UK (June 21-22, 2017) under the title “Assessing drug intervention effects with published FDA approval summary documents: an experience report and practical guidance.” We thank two colleagues at the University Hospital and University of Basel (Dominik Glinz, Basel Institute for Clinical Epidemiology and Biostatistics, and Christopher Traenka, Department of Neurology) for pilot testing the tutorial.
Footnotes
Contributors: AL wrote the first draft with input from LGH, and all authors made critical revisions to the manuscript. All authors read and approved the final version of the paper. LGH and BK obtained funding for this study. AL and LGH are the guarantors.
Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf. All authors declare no financial relationships with any organisation that might have an interest in the submitted work in the previous three years and no other relationships or activities that could appear to have influenced the submitted work.
Funding: This project was supported by the Swiss Cancer League (Grant No KLS-3587-02-2015). The Basel Institute for Clinical Epidemiology and Biostatistics is supported by Stiftung Institut für Klinische Epidemiologie. None of the funders/sponsors had a role in the design and conduct of the project; in the preparation, review, and approval of the manuscript; or the decision to submit the manuscript for publication.
Data sharing: No additional data available.
Transparency: The corresponding author affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.