Original ArticleIn an empirical evaluation of the funnel plot, researchers could not visually identify publication bias
Introduction
Health care providers and policy makers rely on systematic reviews, including meta-analyses, for important decisions concerning clinical care. Yet the conclusions of a review may be invalid if the available studies form a biased sample. Bias occurs because statistically significant research results are more widely disseminated, and therefore easier to retrieve. Unpublished work is the most difficult to access, but selective reporting, frequency of citation, duplicate publication, and language of publication also influence accessibility [1], [2], [3], [4], [5], [6].
Researchers are becoming more aware of the potential for publication and related biases. Many have turned to the funnel plot, a simple graphic technique for detecting these biases [7]. The funnel plot is a scatter plot of the component studies in a review or meta-analysis (quantitative summary), with the treatment effect on the horizontal axis, and a weight, such as the inverse standard error, or sample size, on the vertical axis. Larger weights correspond to more precise estimates of treatment effect. A common interpretation is that a symmetric, inverted funnel shape implies no publication bias, but if the funnel appears to be missing points in the lower corner of the plot associated with ineffectiveness of treatment, there is potential bias (Fig. 1). The funnel shape occurs because the larger, more precise studies tend to be closer to the true effect, whereas the smaller ones are more variable. The simplicity of this method is appealing, and the subjectivity of visual assessment for asymmetry can be overcome by statistically testing for a relation between precision and effect using either regression [8], [9] or rank correlation [10]. However, funnel plots and related statistical methods are problematic because there are several possible causes of funnel plot asymmetry other than publication bias. These include heterogeneity, chance, choice of effect measure, and choice of precision measure [11], [12], [13], [14].
Heterogeneity occurs when results vary from study to study because of differences in study protocol, study quality, illness severity, and patient characteristics. When multiple treatment effects are estimated, there is no reason to expect a funnel shape (Fig. 2).
Although funnel plots are often published or reported for heterogeneous studies [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], the impact of heterogeneity on the shape of the funnel plot is frequently overlooked. For example, in a systematic review of intra-articular hyaluronic acid in treatment of knee osteoarthritis [26], the authors interpreted the funnel plot and regression test as supporting the presence of publication bias (Fig. 3). They recognized molecular weight as a source of heterogeneity, and even plotted the highest molecular weight studies with a different symbol, but they did not take heterogeneity into consideration when assessing for publication bias. Without the high molecular studies, the plot appears more symmetric.
A funnel plot with a small number of studies may appear asymmetric simply due to chance. More than 50% of meta-analyses have 10 studies or fewer [28], [29], and it is not uncommon for researchers to publish interpretations of funnel plots with 10 or fewer points [17], [18], [30], [31], [32], [33], [34], [35], [36], [37], [38].
Inappropriate applications of the funnel plot are not surprising in light of expert advice to assess for publication bias by examining funnel plots for asymmetry [10], [39]. Caveats [11], [12], [13], [14], [28], [39] have generally gone unheeded. One commentary in a subspecialty journal ignored all ambiguity, advising that funnel plots “always” be used, and that a relation between treatment effect and sample size is “indicative of publication bias” [40].
We found previously [14] that a quantitative method of filling in the sparse corner of the funnel plot [41] spuriously adjusted for nonexistent bias if the studies were heterogeneous, or if there were only 10 studies per meta-analysis. We conjectured that visual inspection would also be inadequate for separating the effects of publication bias, heterogeneity, and chance. Many published reports display and visually interpret the funnel plot. The study reported here used a questionnaire to determine whether researchers can correctly identify publication bias from funnel plots, when they are shown both symmetric and asymmetric plots with and without publication bias.
Section snippets
The participants
A questionnaire on funnel plot interpretation was completed in the spring and summer of 2001 by a convenience sample with three groups and a total of 41 participants. Seventeen were taking a meta-analysis course at Tufts University School of Medicine. These were mostly fellows in the Clinical Research Graduate Program of the Sackler School of Graduate Biomedical Sciences. The questionnaires were completed close to the end of the course, but before a class on publication bias. Three more
Results
The mean percentage of plots correctly identified was 52.5% (95% CI, 50.6–54.4%). The means for fellows, faculty, and reviewers (51.5, 53.4, and 53.4%, respectively) were not significantly different. The mean percentage correct, excluding the plots answered “maybe,” was 54.5% (95% CI, 50.9–58.0%) (Fig. 5). Two faculty members are not represented in Fig. 5 because they answered “maybe” to all 16 plots. The variability in the scores decreased as the number answered “yes” or “no” increased. The
Discussion
Medical researchers who participated in the survey tended to interpret funnel plot asymmetry in plots of 10 studies as evidence of publication bias. On average, they identified about 50% of the plots correctly, because at each level of asymmetry, the same number of plots with and without publication bias was included. Although it is not surprising that participants could not distinguish between plots of similar shape with and without publication bias, the results are important because many
Acknowledgments
This work was supported by the Agency for Healthcare Research and Quality (AHRQ), grant number R01 HS10254.
References (52)
- et al.
Publication bias in clinical research
Lancet
(1991) - et al.
Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel bias?
J Clin Epidemiol
(1995) - et al.
Misleading funnel plot for detection of bias in meta-analysis
J Clin Epidemiol
(2000) - et al.
A meta-analysis of the angiotensin-converting enzyme gene polymorphism and restenosis after percutaneous transluminal coronary revascularization: Evidence for publication bias
Am Heart J
(2002) - et al.
Garlic and cancer: a critical review of the epidemiologic literature
J Nutr
(2001) - et al.
Publication and related bias in meta-analysis. Power of statistical tests and prevalence in the literature
J Clin Epidemiol
(2000) - et al.
Effect of fenfluramine-derivative diet pills on cardiac valves: a meta-analysis of observational studies
Am Heart J
(2002) - et al.
Effect of pneumonia case management on mortality in neonates, infants, and preschool children: a meta-analysis of community-based trials
Lancet Infect Dis
(2003) - et al.
Acetylcysteine for prevention of contrast nephropathy: meta-analysis
Lancet
(2003) Quality of reporting of meta-analyses: the QUOROM statement. Will it help?
J Hepatol
(2001)
Summing up evidence: one answer is not always enough
Lancet
Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors
J Clin Epidemiol
The existence of publication bias and risk factors for its occurrence
JAMA
False-positive results in clinical trials: multiple significance tests and the problem of unreported comparisons
J Natl Cancer Inst
Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome
BMJ
Impact of covert duplicate publication on meta-analysis: a case study
BMJ
Summing up: the science of reviewing research
Bias in meta-analysis detected by a simple, graphical test
BJM
A comparison of methods to detect publication bias in meta-analysis
Stat Med
Publication bias
Inflation of type I error rate in two statistical tests for the detection of publication bias in meta-analyses with binary outcomes
Stat Med
Investigating and dealing with publication and other biases
Adjusting for publication bias in the presence of heterogeneity
Stat Med
Publication bias and heterogeneity in the relationship between systolic blood pressure, birth weight, and catch-up growth
J Hypertens
Cisapride treatment for gastro-oesophagaeal reflux in children: a systematic review of randomized controlled trials
J Paediatr Child Health
Fluoride varnishes for preventing dental caries in children and adolescents
Cochrane Database Syst Rev
Cited by (387)
Quadratus Lumborum Block and Transversus Abdominis Plane Block in Non-emergency Cesarean Delivery: A Systematic Review and Meta-analysis
2023, Journal of Perianesthesia Nursing