BMJ  2005;331:433-434 (20 August), doi:10.1136/bmj.38478.497164.F7 (published 3 June 2005)

Paper

Systematic review of publication bias in studies on publication bias

¹ Institut für Allgemeinmedizin, Universitätsklinikum Hamburg-Eppendorf, Martinistrasse 52, 20246 Hamburg, Germany

Introduction

Publication bias is a well known phenomenon in clinical literature,^{1 2} in which positive results have a better chance of being published, are published earlier, and are published in journals with higher impact factors. Conclusions exclusively based on published studies, therefore, can be misleading.³ Selective underreporting of research might be more widespread and more likely to have adverse consequences for patients than publication of deliberately falsified data.¹ We investigated whether there is preferential publication of positive papers on publication bias.

Methods and results

We identified studies that assessed the impact of publication bias in Medline (January 1993 to October 2003) using the search terms "publication bias", "citation bias", "language bias", location bias", "reference bias", or "multiple publication bias". We also searched the references of a Cochrane review on publication bias.⁴ We restricted the search to publications that primarily investigated publication bias and whose acceptance therefore might have depended on whether they had found publication bias or not. We retrieved 265 references. Of these, we chose 148 for full examination. Their bibliographies yielded 26 additional papers. We excluded 148 studies because they gave no original data. All remaining 26 were included in the analysis (see bmj.com).

We used a funnel plot to evaluate reports for publication bias. In a funnel plot the effect size is plotted versus a measure of its precision, such as sample size. With increasing sample size, random variations of the effect are smaller. Thus, data from several studies are expected to be symmetrically distributed in a funnel shaped area of the plot if no publication bias is present. Conversely, an asymmetrical funnel plot indicates a biased study sample.⁵

We plotted effect size versus sample size (figure). The effect is the ratio of the odds of a positive study being published to the odds of a negative study. We transformed reported relative risks into odds ratios. We did not transform hazard ratios. The vertical line indicates no publication bias; 23 of 26 studies report preferential publication of positive results.

View larger version (23K): [in this window] [in a new window]   Funnel plot of 26 reports on publication bias, with reported effect as dependent variable

 

The median reported odds ratio is 2.3, indicating preferred publication of positive results. The sloping line results from a regression with the reported effect as dependent variable. Its slope does not differ significantly from zero (P = 0.13)—that is, the asymmetry of the data is not statistically significant.

Comment

We found no evidence of publication bias in reports on publication bias. But, with just 26 studies, the power to detect asymmetry in a funnel plot was low.⁵ Furthermore, the definition of the terms "positive" and "significant" is non-uniform and sometimes rather arbitrary in the studies reinvestigated here. For example, Dickersin (see bmj.com) used the definition "studies reported to have statistically significant findings were combined with those reported to have findings of great importance. Together they are referred to as `significant' and are contrasted with the remainder, which are referred to as `not significant.'"

Most data on publication bias were recorded retro-spectively and lack prospective registration, as does the present analysis. Prospective and registered studies on publication bias are needed.

What is already known on this topic Studies estimated to have publication bias seem more likely to be published at all, earlier, and in journals with higher impact factors; as a consequence effects are often overestimated What this study adds These findings do not indicate publication bias in reports on publication bias

---

References for the 26 studies are on bmj.com

This article was posted on bmj.com on 3 June 2005: http://bmj.com/cgi/doi/10.1136/bmj.38478.497164.F7

Contributors: Both authors developed the design of this review, did the literature search, and analysed and interpreted the trials. The article was jointly written by both authors. H-HD is guarantor.

Competing interests: None declared.

Funding: Unna Stiftung, Düsseldorf, Germany.

Ethical approval: Not needed.

References

1. Chalmers I. Underreporting research is scientific misconduct. JAMA 1990;263: 1405-8.[Abstract]
2. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337: 867-72.[CrossRef][ISI][Medline]
3. Egger M, Smith GD. Misleading meta-analysis. BMJ 1995;310: 752-4.[Free Full Text]
4. Scherer RW, Langenberg P. Full publication of results initially presented in abstracts. Cochrane Database Methodol Rev 2000;(4): MR000005.
5. Egger M, Smith GD, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997;315: 629-34.[Abstract/Free Full Text]

(Accepted 19 April 2005)

Related Articles

Review of publication bias in studies on publication bias: Studies on publication bias are probably susceptible to the bias they study

Fujian Song
BMJ 2005 331: 637-638. [Extract] [Full Text]

Review of publication bias in studies on publication bias: Meta-research on publication bias does not help transfer research results to patient care

Peter Kranke
BMJ 2005 331: 638. [Extract] [Full Text]

Review of publication bias in studies on publication bias: Here's a proposal for editors that may help reduce publication bias

M Maria Glymour and Ichiro Kawachi
BMJ 2005 331: 638. [Extract] [Full Text]

Review of publication bias in studies on publication bias: Mandatory publication of data may help

Jeffrey Mann
BMJ 2005 331: 638. [Extract] [Full Text]

Review of publication bias in studies on publication bias: Systematic review is needed

Raeburn B Forbes and Mark McCarron
BMJ 2005 331: 638. [Extract] [Full Text]

Studies that find publication bias are no more likely to be published than studies that don't
BMJ 2005 331: 0. [Full Text]

Bias in meta-analysis detected by a simple, graphical test

Matthias Egger, George Davey Smith, Martin Schneider, and Christoph Minder
BMJ 1997 315: 629-634. [Abstract] [Full Text]

Misleading meta-analysis

Matthias Egger and George Davey Smith
BMJ 1995 310: 752-754. [Extract] [Full Text]

This article has been cited by other articles:

• Martin, J. L. R., Sainz-Pardo, M., Furukawa, T. A., Martin-Sanchez, E., Seoane, T., Galan, C. (2007). Review: Benzodiazepines in generalized anxiety disorder: heterogeneity of outcomes based on a systematic review and meta-analysis of clinical trials. J Psychopharmacol 21: 774-782 [Abstract]  
• SINGH, S. P., GREENWOOD, N., WHITE, S., CHURCHILL, R. (2007). Ethnicity and the Mental Health Act 1983. Br. J. Psychiatry 191: 99-105 [Abstract] [Full text]  
• Decullier, E., Chapuis, F. (2007). Oral presentation bias: a retrospective cohort study. J. Epidemiol. Community Health 61: 190-193 [Abstract] [Full text]  
• Kranke, P. (2005). Review of publication bias in studies on publication bias: Meta-research on publication bias does not help transfer research results to patient care. BMJ 331: 638-638 [Full text]  
• Song, F. (2005). Review of publication bias in studies on publication bias: Studies on publication bias are probably susceptible to the bias they study. BMJ 331: 637-638 [Full text]  
• Glymour, M M., Kawachi, I. (2005). Review of publication bias in studies on publication bias: Here's a proposal for editors that may help reduce publication bias. BMJ 331: 638-638 [Full text]  
• Mann, J. (2005). Review of publication bias in studies on publication bias: Mandatory publication of data may help. BMJ 331: 638-638 [Full text]  
• Forbes, R. B, McCarron, M. (2005). Review of publication bias in studies on publication bias: Systematic review is needed. BMJ 331: 638-638 [Full text]  

Rapid Responses:

Read all Rapid Responses

What is publication bias ?

Evelyne Decullier, et al.
bmj.com, 21 Jun 2005 [Full text]

Re: What is publication bias ?

Joseph C Watine
bmj.com, 21 Jun 2005 [Full text]

Re: Re: What is publication bias ?

Evelyne Decullier, et al.
bmj.com, 23 Jun 2005 [Full text]

Re: Re: Re: What is publication bias ?

Joseph C Watine
bmj.com, 28 Jun 2005 [Full text]

Studies on publication bias are likely susceptible to the bias they studied

Fujian Song, et al.
bmj.com, 16 Jul 2005 [Full text]

Meta-research on publication bias does not help transfer research results to patient care

Peter Kranke, MD
bmj.com, 19 Aug 2005 [Full text]

Has Evidence Based Medicine Lost its Way?

Raeburn B Forbes, et al.
bmj.com, 22 Aug 2005 [Full text]

Re: Has Evidence Based Medicine Lost its Way?

Joseph C Watine
bmj.com, 23 Aug 2005 [Full text]

Sparse search strategy

Jafar Kolahi, et al.
bmj.com, 29 Aug 2005 [Full text]

A proposal to reduce publication bias

M. Maria Glymour, et al.
bmj.com, 7 Sep 2005 [Full text]

A less modest proposal

Jeffrey Mann
bmj.com, 8 Sep 2005 [Full text]

Publication bias: Responses and a proposal from the Declaration of Helsinki

Hans-Hermann Dubben
bmj.com, 22 Apr 2006 [Full text]