Original articles
How useful are unpublished data from the Food and Drug Administration in meta-analysis?

https://doi.org/10.1016/S0895-4356(02)00520-6Get rights and content

Abstract

The goals of this systematic review and meta-analysis were to ascertain whether studies of nonsteroidal anti-inflammatory drugs (NSAIDs) summarized in the FDA reviews are ultimately published, to compare the methodologic and population characteristics of studies summarized in the FDA reviews with those reported in peer reviewed literature, and to compare the pooled relative risk of dyspepsia from NSAIDs in each data source. Summary measures of risk difference were calculated with a random effects model; meta-regression was used to assess the effect of study covariates. Among 37 studies described in the FDA reviews, one was published. Sample size, gender distribution, indication for drug use, and methodologic quality did not vary significantly between the published and FDA data. The pooled risk ratio for dyspepsia obtained using published data (1.21) or FDA data (1.07) did not differ significantly or practically. Data from FDA reviews may be a viable data source for systematic reviews and meta-analyses but only after being subjected to the same methodologic scrutiny as published data.

Introduction

The inclusion of unpublished data in systematic reviews or meta-analyses is controversial 1, 2. Proponents argue that exclusion of unpublished data may systematically alter the conclusions of meta-analyses, and therefore attempts should be made to include all available evidence. Opponents argue that unpublished data have not gone through the crucible of the peer-review process and are therefore more suspect than published data. One large survey concluded that most experts believe unpublished data should be included as long as the studies can be subjected to the same scrutiny as published data [1]. However, finding unpublished evidence presents a challenge because there does not exist a systematic way of accessing it.

One potentially large and easily accessible systematic repository of unpublished data on the efficacy and safety of pharmaceuticals is the United States Food and Drug Administration (FDA), which produces publicly available reviews of all studies submitted as part of the New Drug Application (NDA) process. However, the content and methodologic quality of these data have not been evaluated, nor has there been an assessment of how many studies in the FDA reviews also appear in the peer-reviewed literature (ie, the extent to which these data are “unpublished”). We offer the first such look, which we undertook as part of a meta-analysis assessing the complications of nonsteroidal anti-inflammatory drugs (NSAIDs).

Section snippets

Methods

As part of the regulatory process to approve drugs for use in the United States, sponsors must submit a NDA to the FDA. NDAs contain extensive information about studies conducted to support claims about the safety and efficacy of the drug for its proposed indications. FDA medical officers review these data and prepare reports summarizing that information that are subsequently used to evaluate the merits of the NDA. The original study material is considered proprietary and not available to the

Studies identified

We identified 4881 published titles that included the name of an NSAID or pertained to NSAIDs as a group (Figure 1). Among these, 2704 were rejected because the title clearly did not describe studies of NSAIDs among adults. An additional 377 articles were rejected after reviewing the abstract because they did not describe studies of NSAIDs among adults. Among the remaining 1800 articles, 1730 (96%) full text articles were obtained and screened. Among the articles screened, 1397 were rejected

Discussion

Our results provide some measure of reassurance to people who are concerned about the methodologic quality of manufacturer-supported studies submitted to the FDA as part of the NDA process. We found no meaningful difference between published studies (most of which were manufacturer sponsored) and manufacturer-sponsored unpublished studies on the most widely accepted criteria for methodologic quality of controlled trials, the Jadad scale. Although more published than FDA studies reported

Conclusions

Data from FDA reviews should not be excluded from systematic reviews and meta-analyses based solely on concerns about their methodologic integrity. Rather, these data can be considered a viable source for systematic reviews and meta-analyses but only after being subjected to the same methodologic scrutiny as published data.

References (32)

  • M Angell

    Negative studies

    New Engl J Med

    (1989)
  • Data on file. Westport, CT: IMS...
  • K.F Schulz et al.

    Dimensions of methodological quality associated with estimates of treatment effects in controlled trials

    JAMA

    (1995)
  • K.S Khan et al.

    The importance of quality of primary studies in producing unbiased systematic reviews

    Arch Intern Med

    (1996)
  • P.A Rochon et al.

    A study of manufacturer-supported trials of nonsteroidal anti-inflammatory drugs in the treatment of arthritis

    Arch Intern Med

    (1994)
  • L.J Cronbach

    Coefficient alpha and the internal structure of tests

    Psychometrika

    (1951)
  • Cited by (0)

    Supported by a contract from Merck & Co. to RAND.

    Dr. Shekelle is a Senior Research Associate of the Veterans Affairs Health Services Research and Development.

    View full text