GRADE Series - Sharon Straus, Rachel Churchill and Sasha Shepperd, Guest Editors
GRADE guidelines: 5. Rating the quality of evidence—publication bias

https://doi.org/10.1016/j.jclinepi.2011.01.011Get rights and content

Abstract

In the GRADE approach, randomized trials start as high-quality evidence and observational studies as low-quality evidence, but both can be rated down if a body of evidence is associated with a high risk of publication bias. Even when individual studies included in best-evidence summaries have a low risk of bias, publication bias can result in substantial overestimates of effect. Authors should suspect publication bias when available evidence comes from a number of small studies, most of which have been commercially funded. A number of approaches based on examination of the pattern of data are available to help assess publication bias. The most popular of these is the funnel plot; all, however, have substantial limitations. Publication bias is likely frequent, and caution in the face of early results, particularly with small sample size and number of events, is warranted.

Introduction

Key points

  • Empirical evidence shows that, in general, studies with statistically significant results are more likely to be published than studies without statistically significant results (“negative studies”).

  • Systematic reviews performed early, when only few initial studies are available, will overestimate effects when “negative” studies face delayed publication. Early positive studies, particularly if small in size, are suspect.

  • Recent revelations suggest that withholding of “negative” results by industry sponsors is common. Authors of systematic reviews should suspect publication bias when studies are uniformly small, particularly when sponsored by the industry.

  • Empirical examination of patterns of results (e.g., funnel plots) may suggest publication bias but should be interpreted with caution.

In four previous articles in our series describing the GRADE system of rating the quality of evidence and grading the strength of recommendations, we have described the process of framing the question, introduced GRADE’s approach to rating the quality of evidence, and dealt with the possibility of rating down quality for study limitations (risk of bias). This fifth article deals with the another of the five categories of reasons for rating down the quality of evidence: publication bias. Our exposition relies to some extent on prior work addressing issues related to publication bias [1]; we did not conduct a systematic review of the literature relating to publication bias.

Even if individual studies are perfectly designed and executed, syntheses of studies may provide biased estimates because systematic review authors or guideline developers fail to identify studies. In theory, the unidentified studies may yield systematically larger or smaller estimates of beneficial effects than those identified. In practice, there is more often a problem with “negative” studies, the omission of which leads to an upward bias in estimate of effect. Failure to identify studies is typically a result of studies remaining unpublished or obscurely published (e.g., as abstracts or theses)—thus, methodologists have labeled the phenomenon “publication bias.”

An informative systematic review assessed the extent to which publication of a cohort of clinical trials is influenced by the statistical significance, perceived importance, or direction of their results [2]. It found five studies that investigated these associations in a cohort of registered clinical trials. Trials with positive findings were more likely to be published than trials with negative or null findings (odds ratio: 3.90; 95% confidence interval [CI]: 2.68, 5.68). This corresponds to a risk ratio of 1.78 (95% CI: 1.58, 1.95), assuming that 41% of negative trials are published (the median among the included studies, range = 11–85%). In absolute terms, this means that if 41% of negative trials are published, we would expect that 73% of positive trials would be published. Two studies assessed time to publication and showed that trials with positive findings tended to be published after 4–5 years compared with those with negative findings, which were published after 6–8 years. Three studies found no statistically significant association between sample size and publication. One study found no statistically significant association between either funding mechanism, investigator rank, or sex and publication.

Section snippets

Publication bias vs. selective reporting bias

In some classification systems, reporting bias has two subcategories: selective outcome reporting, with which we have dealt in the previous article in the series, and publication bias. However, all the sources of bias that we have considered under study limitations, including selective outcome reporting, can be addressed in single studies. In contrast, when an entire study remains unreported and reporting is related to the size of the effect—publication bias—one can assess the likelihood of

Variations in publication bias

The results of a systematic review will be biased if the sample of studies included is unrepresentative—whether the studies not included are published or not. Thus, biased conclusions can result from an early review that omits studies with delayed publication—a phenomenon sometimes termed “lag bias” [8]. Either because authors do not submit studies with what they perceive as uninteresting results to prominent journals or because of repeated rejection at such journals, a study may end up

Bigger dangers of publication bias in reviews with small studies

The risk of publication bias may be higher for reviews that are based on small randomized controlled trials (RCTs) [17], [18], [19]. RCTs including large numbers of patients are less likely to remain unpublished or ignored and tend to provide more precise estimates of the treatment effect, whether positive or negative (i.e., showing or not showing a statistically significant difference between intervention and control groups). Discrepancies between results of meta-analyses of small studies and

Large studies are not immune

Although large studies are more likely to be published, sponsors who are displeased with the results may delay or even suppress publication [14], [22], [23]. Furthermore, they may publish in journals with limited readership studies that, by their significance, warrant publication in the highest profile medical journals. They may also succeed in obscuring results using strategies that are scientifically unsound. The following example illustrates all these phenomena.

Salmeterol Multicentre Asthma

When to rate down for publication bias—industry influence

In general, review authors and guideline developers should consider rating down for likelihood of publication bias when the evidence consists of a number of small studies [17], [18], [19], [20], [21]. The inclination to rate down for publication bias should increase if most of those small studies are industry sponsored or likely to be industry sponsored (or if the investigators share another conflict of interest) [14], [23], [28].

An investigation of 74 antidepressant trials with a mean sample

Using study results to estimate the likelihood of publication bias

Another criterion for publication bias is the pattern of study results. Suspicion may increase if visual inspection demonstrates an asymmetrical (Fig. 1b) rather than a symmetrical (Fig. 1a) funnel plot or if statistical tests of asymmetry are positive [29], [30]. Although funnel plots may be helpful, review authors and guideline developers should bear in mind that visual assessment of funnel plots is distressingly prone to error [31], [32]. Enhancements of funnel plots may (or may not) help to

Publication bias in observational studies

The risk of publication bias is probably larger for observational studies than for RCTs [3], [32], particularly small observational studies and studies conducted on data collected automatically (e.g., in the electronic medical record or in a diabetes registry) or data collected for a previous study. In these instances, it is difficult for the reviewer to know if the observational studies that appear in the literature represent all or a fraction of the studies conducted, and whether the analyses

Rating down for publication bias—an example

A systematic review of flavonoids in patients with hemorrhoids provides an example of a body of evidence in which rating down for publication bias is likely appropriate [48]. All trials, which ranged in size from 40 to 234 patients—with most around 100—were industry sponsored. Furthermore, the funnel plot suggests the possibility of publication bias (Fig. 2).

Acknowledging the difficulties in assessing the likelihood of publication bias

Unfortunately, it is very difficult to be confident that publication bias is absent, and almost equally difficult to know where to place the threshold and rate down for its likely presence. Recognizing these challenges, the terms GRADE suggests using in GRADE evidence profiles for publication bias are “undetected” and “strongly suspected.” Acknowledging the uncertainty, GRADE suggests rating down a maximum of one level (rather than two) for suspicion of publication bias. Nevertheless, the

References (48)

  • S. Hopewell et al.

    Publication bias in clinical trials due to statistical significance or direction of trial results

    Cochrane Database Syst Rev

    (2009)
  • K. Dickersin et al.

    Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards

    JAMA

    (1992)
  • J.M. Stern et al.

    Publication bias: evidence of delayed publication in a cohort study of clinical research projects

    BMJ

    (1997)
  • A.H. Bardy

    Bias in reporting clinical trials

    Br J Clin Pharmacol

    (1998)
  • M. Egger et al.

    Bias in location and selection of studies

    BMJ

    (1998)
  • F. Song et al.

    Publication and related biases

    Health Technol Assess

    (2000)
  • S. Hopewell et al.

    Time to publication for results of clinical trials

    Cochrane Database Syst Rev

    (2008)
  • P. Juni et al.

    Direction and impact of language bias in meta-analyses of controlled trials: empirical study

    Int J Epidemiol

    (2002)
  • S. Hopewell et al.

    Grey literature in meta-analyses of randomized trials of health care interventions

    Cochrane Database Syst Rev

    (2007)
  • D. Rennie

    Fair conduct and fair reporting of clinical trials

    JAMA

    (1999)
  • M.R. Tramer et al.

    Impact of covert duplicate publication on meta-analysis: a case study

    BMJ

    (1997)
  • H. Melander et al.

    Evidence b(i)ased medicine—selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications

    BMJ

    (2003)
  • E. von Elm

    Different patterns of duplicate publication: an analysis of articles used in systematic reviews

    JAMA

    (2004)
  • C.B. Begg et al.

    Publication bias and dissemination of clinical research

    J Natl Cancer Inst

    (1989)
  • Cited by (1279)

    • Preventing postpartum hemorrhage: A network meta-analysis on routes of administration of uterotonics

      2024, European Journal of Obstetrics and Gynecology and Reproductive Biology
    View all citing articles on Scopus

    The GRADE system has been developed by the GRADE Working Group. The named authors drafted and revised this article. A complete list of contributors to this series can be found on the journal's Web site at www.elsevier.com.

    View full text