Original Article
In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias

https://doi.org/10.1016/j.jclinepi.2005.01.006Get rights and content

Abstract

Background and Objective

Publication bias and related biases can lead to overly optimistic conclusions in systematic reviews. The funnel plot, which is frequently used to detect such biases, has not yet been subjected to empirical evaluation as a visual tool. We sought to determine whether researchers can correctly identify publication bias from visual inspection of funnel plots in typical-size systematic reviews.

Methods

A questionnaire with funnel plots containing 10 studies each (the median number in medical meta-analyses) was completed by 41 medical researchers, including clinical research fellows in a meta-analysis class, faculty in clinical care research, and experienced systematic reviewers.

Results

On average, participants correctly identified 52.5% (95% CI 50.6–54.4%) of the plots as being affected or unaffected by publication bias. The weighted mean percent correct, which adjusted for the fact that asymmetric plots are more likely to occur in the presence of publication bias, was also low (48.3 to 62.8%, depending on the presence or absence of publication bias and heterogeneous study effects).

Conclusion

Researchers who assess for publication bias using the funnel plot may be misled by its shape. Authors and readers of systematic reviews need to be aware of the limitations of the funnel plot.

Introduction

Health care providers and policy makers rely on systematic reviews, including meta-analyses, for important decisions concerning clinical care. Yet the conclusions of a review may be invalid if the available studies form a biased sample. Bias occurs because statistically significant research results are more widely disseminated, and therefore easier to retrieve. Unpublished work is the most difficult to access, but selective reporting, frequency of citation, duplicate publication, and language of publication also influence accessibility [1], [2], [3], [4], [5], [6].

Researchers are becoming more aware of the potential for publication and related biases. Many have turned to the funnel plot, a simple graphic technique for detecting these biases [7]. The funnel plot is a scatter plot of the component studies in a review or meta-analysis (quantitative summary), with the treatment effect on the horizontal axis, and a weight, such as the inverse standard error, or sample size, on the vertical axis. Larger weights correspond to more precise estimates of treatment effect. A common interpretation is that a symmetric, inverted funnel shape implies no publication bias, but if the funnel appears to be missing points in the lower corner of the plot associated with ineffectiveness of treatment, there is potential bias (Fig. 1). The funnel shape occurs because the larger, more precise studies tend to be closer to the true effect, whereas the smaller ones are more variable. The simplicity of this method is appealing, and the subjectivity of visual assessment for asymmetry can be overcome by statistically testing for a relation between precision and effect using either regression [8], [9] or rank correlation [10]. However, funnel plots and related statistical methods are problematic because there are several possible causes of funnel plot asymmetry other than publication bias. These include heterogeneity, chance, choice of effect measure, and choice of precision measure [11], [12], [13], [14].

Heterogeneity occurs when results vary from study to study because of differences in study protocol, study quality, illness severity, and patient characteristics. When multiple treatment effects are estimated, there is no reason to expect a funnel shape (Fig. 2).

Although funnel plots are often published or reported for heterogeneous studies [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], the impact of heterogeneity on the shape of the funnel plot is frequently overlooked. For example, in a systematic review of intra-articular hyaluronic acid in treatment of knee osteoarthritis [26], the authors interpreted the funnel plot and regression test as supporting the presence of publication bias (Fig. 3). They recognized molecular weight as a source of heterogeneity, and even plotted the highest molecular weight studies with a different symbol, but they did not take heterogeneity into consideration when assessing for publication bias. Without the high molecular studies, the plot appears more symmetric.

A funnel plot with a small number of studies may appear asymmetric simply due to chance. More than 50% of meta-analyses have 10 studies or fewer [28], [29], and it is not uncommon for researchers to publish interpretations of funnel plots with 10 or fewer points [17], [18], [30], [31], [32], [33], [34], [35], [36], [37], [38].

Inappropriate applications of the funnel plot are not surprising in light of expert advice to assess for publication bias by examining funnel plots for asymmetry [10], [39]. Caveats [11], [12], [13], [14], [28], [39] have generally gone unheeded. One commentary in a subspecialty journal ignored all ambiguity, advising that funnel plots “always” be used, and that a relation between treatment effect and sample size is “indicative of publication bias” [40].

We found previously [14] that a quantitative method of filling in the sparse corner of the funnel plot [41] spuriously adjusted for nonexistent bias if the studies were heterogeneous, or if there were only 10 studies per meta-analysis. We conjectured that visual inspection would also be inadequate for separating the effects of publication bias, heterogeneity, and chance. Many published reports display and visually interpret the funnel plot. The study reported here used a questionnaire to determine whether researchers can correctly identify publication bias from funnel plots, when they are shown both symmetric and asymmetric plots with and without publication bias.

Section snippets

The participants

A questionnaire on funnel plot interpretation was completed in the spring and summer of 2001 by a convenience sample with three groups and a total of 41 participants. Seventeen were taking a meta-analysis course at Tufts University School of Medicine. These were mostly fellows in the Clinical Research Graduate Program of the Sackler School of Graduate Biomedical Sciences. The questionnaires were completed close to the end of the course, but before a class on publication bias. Three more

Results

The mean percentage of plots correctly identified was 52.5% (95% CI, 50.6–54.4%). The means for fellows, faculty, and reviewers (51.5, 53.4, and 53.4%, respectively) were not significantly different. The mean percentage correct, excluding the plots answered “maybe,” was 54.5% (95% CI, 50.9–58.0%) (Fig. 5). Two faculty members are not represented in Fig. 5 because they answered “maybe” to all 16 plots. The variability in the scores decreased as the number answered “yes” or “no” increased. The

Discussion

Medical researchers who participated in the survey tended to interpret funnel plot asymmetry in plots of 10 studies as evidence of publication bias. On average, they identified about 50% of the plots correctly, because at each level of asymmetry, the same number of plots with and without publication bias was included. Although it is not surprising that participants could not distinguish between plots of similar shape with and without publication bias, the results are important because many

Acknowledgments

This work was supported by the Agency for Healthcare Research and Quality (AHRQ), grant number R01 HS10254.

References (52)

  • J. Lau et al.

    Summing up evidence: one answer is not always enough

    Lancet

    (1998)
  • C.H. Schmid et al.

    Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors

    J Clin Epidemiol

    (2004)
  • K. Dickersin

    The existence of publication bias and risk factors for its occurrence

    JAMA

    (1990)
  • I.F. Tannock

    False-positive results in clinical trials: multiple significance tests and the problem of unreported comparisons

    J Natl Cancer Inst

    (1996)
  • U. Ravnskov

    Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome

    BMJ

    (1992)
  • M.R. Tramer et al.

    Impact of covert duplicate publication on meta-analysis: a case study

    BMJ

    (1997)
  • R.J. Light et al.

    Summing up: the science of reviewing research

    (1984)
  • M. Egger et al.

    Bias in meta-analysis detected by a simple, graphical test

    BJM

    (1997)
  • P. Macaskill et al.

    A comparison of methods to detect publication bias in meta-analysis

    Stat Med

    (2001)
  • C.B. Begg

    Publication bias

  • G. Schwarzer et al.

    Inflation of type I error rate in two statistical tests for the detection of publication bias in meta-analyses with binary outcomes

    Stat Med

    (2002)
  • J.A.C. Sterne et al.

    Investigating and dealing with publication and other biases

  • N. Terrin et al.

    Adjusting for publication bias in the presence of heterogeneity

    Stat Med

    (2003)
  • M.D. Schluchter

    Publication bias and heterogeneity in the relationship between systolic blood pressure, birth weight, and catch-up growth

    J Hypertens

    (2003)
  • R.E. Gilbert et al.

    Cisapride treatment for gastro-oesophagaeal reflux in children: a systematic review of randomized controlled trials

    J Paediatr Child Health

    (2000)
  • V.C. Marinho et al.

    Fluoride varnishes for preventing dental caries in children and adolescents

    Cochrane Database Syst Rev

    (2003)
  • Cited by (387)

    View all citing articles on Scopus
    View full text