Review Article
Uptake of methods to deal with publication bias in systematic reviews has increased over time, but there is still much scope for improvement

https://doi.org/10.1016/j.jclinepi.2010.04.022Get rights and content

Abstract

Objective

To evaluate the measures taken to deal with publication bias across different categories of systematic reviews published in 2006 and to compare these with reviews published in 1996.

Study Design and Setting

PubMed was searched for systematic reviews published in 2006; 100 treatment effect, 50 diagnostic accuracy, 100 risk factor, and 50 gene–disease association reviews were randomly selected.

Results

The use of MEDLINE increased from 74% to 95%; checking references increased from 42% to 73%; use of Cochrane Library increased from 5% to 58%; and use of CINAHL increased from 8% in 1996 to 24% in treatment reviews, 20% in diagnostic reviews, 18% in risk factor reviews, and 0% in genetic reviews published in 2006. A 20% increase was observed for explicit searching of non–English-language studies in all reviews published in 2006. Efforts to search for unpublished studies increased to 61% from 35% in treatment reviews published in 1996. Twenty-six percent of the reviews used funnel plots or related methods to test for publication bias compared with less than 6% in earlier reviews.

Conclusion

Recent reviews show a significant improvement in the measures taken to prevent publication bias. However, few methods exist to deal with publication bias in the nonquantitative findings of systematic reviews.

Introduction

What is new?

  • In this study, we assessed the measures taken to deal with publication bias in systematic reviews published in 2006 and compared them with those used in reviews published in 1996. Systematic reviewers are searching more widely, including for non-English (39% vs. 19% in 1996) and unpublished studies (61% vs. 35% in earlier reviews).

  • Reviews also more frequently characterize the presence of publication bias. Twenty-six percent of the reviews used funnel plots and related methods to test for publication bias in 2006 compared with less than 6% in 1996 reviews.

  • Further research is required to develop and validate methods for the assessment of the risk of publication bias in nonquantitative aspects of systematic reviews.

Publication bias has been defined as “the tendency on the part of investigators, reviewers, and editors to submit or accept manuscripts for publication based on the direction or strength of the study findings” [1]. Undetected publication bias can lead to inaccurate estimates of treatment effect sizes, with serious health care implications. If the results of a clinical trial are not published because the intervention was found to be harmful, there is a risk that other researchers may unknowingly cause greater harm by continuing to conduct studies using that intervention or even include the intervention in routine health care. Systematic reviews containing a nonrepresentative selection of the data may, therefore, undermine the validity of clinical guidelines and practice; hence, it is important that researchers, clinicians, and health policy decision makers are aware of the risk of publication bias.

Empirical evidence suggests that publication bias is widespread and has led to the development of methods to identify and minimize such biases [2]. These methods can be sequentially classified as follows: preventing publication bias occurring before a systematic review (e.g., prospective registration of trials); reducing publication bias occurring during a systematic review (e.g., locating unpublished or gray literature studies); detecting publication bias during a systematic review (e.g., funnel plot and related tests, sensitivity analysis modeling); and minimizing the impact of publication bias after a systematic review (e.g., confirmatory large-scale trials, updating the systematic review) [2].

A Health Technology Assessment (HTA) report published in 2000 assessed 193 systematic reviews published in 1996 to examine the issues and methods pertinent to publication bias [3]. This assessment concluded that publication bias had been largely ignored, and that available methods for dealing with publication bias had not been used in most systematic reviews. It also suggested that literature searching was inadequate in some reviews [4]. However, this assessment of reviews published in 1996 had several shortcomings. First, all these systematic reviews were included in the Database of Reviews of Effectiveness and were of superior quality compared with those from general bibliographic databases. Hence, the representativeness of the sample was questionable (implying that rates of uptake may actually have been even lower). Second, reviews on effectiveness of health care interventions and accuracy of diagnostic technologies were not assessed separately to determine if there are differences in methods used to address publication bias between different types of reviews.

In this article, we sought to examine methods used to deal with publication bias in a sample of systematic reviews published in 2006 and to assess the overall risk of publication bias across different categories of reviews.

Section snippets

Literature search

PubMed was searched using the keywords “systematic review” or “meta-analysis,” limited to titles or abstracts, and English-language articles published in 2006. This search yielded 3,503 references, which were assessed for inclusion and then categorized as reviews of treatment effectiveness, reviews of diagnostic accuracy, reviews of risk factors, or reviews of gene–disease association by one reviewer (S.P.B.). The first 100 selected references were also checked for agreement on categorization

Main characteristics of included reviews

The 100 treatment effectiveness reviews comprised 54% reviews of pharmaceutical interventions, 10% reviews of psychological interventions, 11% reviews of surgical interventions, and 25% reviews categorized as mixed or other interventions. The median number of individual studies included in each review was 14 (range: 2–198).

The tests or techniques investigated in the 50 reviews of diagnostic accuracy comprised laboratory tests (42%), imaging techniques (56%), electrical tests (10%), clinical

Discussion

The previous assessment of systematic reviews published in 1996 concluded that the issue of publication bias was largely being overlooked, with very few reviews actually using any methods to prevent, identify, or counter publication bias [3]. The assessment of a sample of systematic reviews published in 2006 enabled us to compare changes over time. There has been a marked increase in the use of measures taken to reduce and identify publication bias in systematic reviews in the 10 years between

Conclusions

Our study suggests that authors of systematic reviews have become more aware of the issues pertaining to publication and related biases since 1996 and are better both at searching to prevent publication bias and at identifying residual publication bias. They are making greater efforts to minimize publication bias in treatment reviews compared with diagnostic, risk factor, and genetics reviews published in 2006, although treatment, diagnostic, and risk factor reviews are less likely to use

Acknowledgments

This methodological review was funded by the UK Department of Health HTA Research Methodology Programme (Code 06/92/02). The views expressed in this report are those of the authors and do not necessarily reflect those of the funders. The authors take the responsibility for any errors. The authors would like to thank Julie Reynolds for providing secretarial support to this review.

References (25)

  • R. Lowry
  • S. McDonald et al.

    Searching the right database. A comparison of four databases for psychiatry journals

    Health Libr Rev

    (1999)
  • Cited by (44)

    • Registration and Reporting Quality of Systematic Reviews on Surgical Intervention: A Meta-Epidemiological Study

      2022, Journal of Surgical Research
      Citation Excerpt :

      Secondly, only 61.5% of registered SRs and 60.0% of non-registered SRs reported publication bias. Publication bias is a well-recognized threat to the validity of SRs.28,29 Authors conducting SRs should assess the likelihood and severity of publication bias or should, alternatively, provide valid reasons for not undertaking such an assessment.

    • Self-conductive interventions by educators aiming to promote individual occupational well-being—A systematic review

      2021, International Journal of Educational Research
      Citation Excerpt :

      Excluding criteria at this stage was done on the basis of the following: 1) wrong study design, 2) wrong population, or 3) wrong outcome. The screening was implemented blinded in the electronic, Rayyan-software, to lessen the risk of selection bias (Parekh-Bhurke et al., 2011). The final stage of the independent reading (J.R. & S.K.) was the reading of the full texts of the 57 studies, of which 13 were selected for this review thus excluding 44 articles for following reasons: 1) wrong study design, 2) wrong outcome interest, 3) wrong population, 4) not self-conductible and 5) was not implemented during working hours.

    • Publication Bias

      2019, Encyclopedia of Pharmacy Practice and Clinical Pharmacy: Volumes 1-3
    View all citing articles on Scopus

    Competing interest: none declared.

    Authors' contributions: F.S., Y.K.L., L.H., J.J.R., A.J.S., and I.H. developed the review protocol. S.P.B., J.J.R., and F.S. conducted literature search. S.P.B., F.S., L.H., Y.K.L., J.J.R., C.B.H., C.S.K., and C.P. extracted and/or checked data from included studies. F.S., A.J.S., and I.H. provided methodological support. S.P.B., F.S., C.S.K., and C.P. analyzed data and drafted the article. All authors commented on the final manuscript.

    View full text