Introduction

Publication bias is a well known phenomenon in clinical literature,1 2 in which positive results have a better chance of being published, are published earlier, and are published in journals with higher impact factors. Conclusions exclusively based on published studies, therefore, can be misleading.3 Selective underreporting of research might be more widespread and more likely to have adverse consequences for patients than publication of deliberately falsified data.1 We investigated whether there is preferential publication of positive papers on publication bias.

Methods and results

We identified studies that assessed the impact of publication bias in Medline (January 1993 to October 2003) using the search terms “publication bias”, “citation bias”, “language bias”, location bias”, “reference bias”, or “multiple publication bias”. We also searched the references of a Cochrane review on publication bias.4 We restricted the search to publications that primarily investigated publication bias and whose acceptance therefore might have depended on whether they had found publication bias or not. We retrieved 265 references. Of these, we chose 148 for full examination. Their bibliographies yielded 26 additional papers. We excluded 148 studies because they gave no original data. All remaining 26 were included in the analysis (see bmj.com).

We used a funnel plot to evaluate reports for publication bias. In a funnel plot the effect size is plotted versus a measure of its precision, such as sample size. With increasing sample size, random variations of the effect are smaller. Thus, data from several studies are expected to be symmetrically distributed in a funnel shaped area of the plot if no publication bias is present. Conversely, an asymmetrical funnel plot indicates a biased study sample.5

We plotted effect size versus sample size (figure). The effect is the ratio of the odds of a positive study being published to the odds of a negative study. We transformed reported relative risks into odds ratios. We did not transform hazard ratios. The vertical line indicates no publication bias; 23 of 26 studies report preferential publication of positive results.

View larger version:

• In a new window

• Download as PowerPoint Slide

Funnel plot of 26 reports on publication bias, with reported effect as dependent variable

The median reported odds ratio is 2.3, indicating preferred publication of positive results. The sloping line results from a regression with the reported effect as dependent variable. Its slope does not differ significantly from zero (P = 0.13)—that is, the asymmetry of the data is not statistically significant.

Comment

We found no evidence of publication bias in reports on publication bias. But, with just 26 studies, the power to detect asymmetry in a funnel plot was low.5 Furthermore, the definition of the terms “positive” and “significant” is non-uniform and sometimes rather arbitrary in the studies reinvestigated here. For example, Dickersin (see bmj.com) used the definition “studies reported to have statistically significant findings were combined with those reported to have findings of great importance. Together they are referred to as ‘significant’ and are contrasted with the remainder, which are referred to as ‘not significant.’”

Most data on publication bias were recorded retro-spectively and lack prospective registration, as does the present analysis. Prospective and registered studies on publication bias are needed.