Endgames Statistical Question

What is publication bias in a meta-analysis?

BMJ 2015; 351 doi: https://doi.org/10.1136/bmj.h4419 (Published 14 August 2015) Cite this as: BMJ 2015;351:h4419
  1. Philip Sedgwick , reader in medical statistics and medical education
  1. 1Institute for Medical and Biomedical Education, St George’s, University of London, London, UK
  1. Correspondence to: P Sedgwick p.sedgwick{at}sgul.ac.uk

Researchers undertook a meta-analysis of the effectiveness of non-benzodiazepine hypnotics, including eszopiclone, zaleplon, and zolpidem. Randomised controlled trials were included if they were double blind, placebo controlled, and had parallel treatment groups. Participants were adults with primary insomnia (transient or chronic). The main outcomes were polysomnographic and subjective measurements of sleep latency.1

Thirteen trials were eligible for inclusion. When the results of the trials were combined, non-benzodiazepine hypnotics showed a significant improvement (reduction) in the main outcome of polysomnographic sleep latency compared with placebo (weighted mean raw difference −22.0 minutes, 95% confidence interval −33.0 to −11.0). Non-benzodiazepine hypnotics showed a small improvement in subjective sleep latency although the change was not significant (−6.9 minutes, −26.0 to 12.4). Improvements in sleep latency did not vary between individual hypnotics. The researchers also reported that Egger’s test showed no evidence that the results were significantly affected by publication bias.

Which of the following, if any, would have resulted in publication bias?

  • a) Trials could not be included in the meta-analysis because they were not published owing to non-significant treatment effects

  • b) Trials reporting significant treatment effects were cited more often in publications, increasing the likelihood of being identified and included in the meta-analysis

  • c) Trials were not included because they were published in inaccessible languages

  • d) Inclusion of a trial because the topic was of interest or importance so its publication was rushed

Answers

Answer a would have resulted in publication bias, whereas answers b, c, and d would not have.

The aim of the meta-analysis was to investigate the effectiveness of the non-benzodiazepine hypnotics—eszopiclone, zaleplon, and zolpidem. The main outcomes were polysomnographic and subjective measurements of sleep latency. The purpose of the meta-analysis was to combine the sample estimates of the treatment effects from each trial that was identified and to give a total estimate for each outcome, thereby reducing a large amount of information to a manageable quantity.

The researchers may not have identified all the relevant trials that had been conducted. If so, it might mean that the total overall estimates produced by the meta-analysis were biased and therefore misleading. If not all the relevant trials are identified in a meta-analysis, the effect of an intervention will probably be overestimated. Failure to include all relevant trials can be the result of various types of bias, described collectively as reporting biases. Reporting biases occur when the reporting of research findings is influenced by the nature and direction of the trial results.

Publication bias is the most well known reporting bias. It results from the publication or non-publication of relevant trials, depending on the nature and direction of the results. For example, a study is more likely to be published if the results are significant. Trials with treatment effects that are not significant are less likely to be published, with the result that authors of meta-analyses and systematic reviews will fail to include all of the relevant studies (answer a).

As described, various biases can contribute to the inclusion or exclusion of a study in a meta-analysis. Such biases are described collectively as reporting biases, and publication bias is just one of them. Failure to include in a meta-analysis all of the relevant studies that have been conducted is often wrongly attributed solely to publication bias. Furthermore, reporting bias is often incorrectly referred to as publication bias. Reporting biases also include citation bias, language bias, and time lag bias. These are described further below. The presence of reporting biases can be detected by a graphical test—a funnel plot—or by using formal statistical tests such as Egger’s test. These methods will be described in a future question. In the above meta-analysis the researchers tested for reporting biases using Egger’s test, but they wrongly indicated that the purpose of the test was to test for publication bias alone.

When performing the meta-analysis, the reference lists of those trials identified would have been examined for other potential trials. Trials may have been cited because of the nature and direction of their results—for example, if non-benzodiazepine hypnotics were shown to be superior to placebo. If so, this would have led to citation bias—that is, studies being cited more often, increasing the likelihood of the trial being identified and included in the meta-analysis (answer b).

Most research papers are written in English, particularly those published in prominent journals. However, members of the research team may not have been able to translate articles published in other languages. Therefore, studies published in languages other than English would probably not have been included in the meta-analysis. If so, language bias would have occurred—that is, the selective inclusion of studies in the meta-analysis because they were published in an accessible language (answer c).

The inclusion of a trial in a meta-analysis may be influenced by whether the publication of the study was rushed or delayed because of the nature and direction of the results. For example, if the topic was of great medical interest or importance then the paper would probably be published sooner. However, if the results of a trial were inconclusive its publication might be delayed. Publication may also be delayed if the paper is not accepted by the researchers’ journal of first choice. If so, time lag bias would have occurred—that is, inclusion of a trial in the meta-analysis would be influenced by the rushed or delayed publication of the study (answer d).

In addition to the reporting biases described above, other types of bias might have influenced whether a trial was included in the meta-analysis. Trials might have been published in journals or databases that are not easily accessible. Referred to as the “grey literature,” this includes abstracts of research presented at conferences and published in conference proceedings. Furthermore, the publication of a trial could have been influenced by who funded or sponsored the study, or even the identity of the research group. Multiple publication bias would have occurred if a trial was published more than once, increasing the likelihood of the trial being identified and included in the meta-analysis. Such a bias might occur if, for example, a trial showed that non-benzodiazepine hypnotics were superior to placebo. Location bias would have been present if trials were published in journals located in medical journal databases that were easier to access. Outcome reporting bias would have been present if researchers selected which of the measured outcomes were reported in the publication. Such a bias might occur if the results of a trial were not as expected or were undesirable.

Notes

Cite this as: BMJ 2015;351:h4419

Footnotes

  • Competing interest: None declared.

References

View Abstract

Sign in

Log in through your institution

Subscribe