Intended for healthcare professionals

Letters Impact of outcome reporting bias

Do systematic reviews still exclude studies with “no relevant outcome data”?

BMJ 2017; 358 doi: https://doi.org/10.1136/bmj.j3919 (Published 21 August 2017) Cite this as: BMJ 2017;358:j3919
  1. Kerry M Dwan, statistical editor1,
  2. Paula R Williamson, professor of biostatistics2,
  3. Jamie J Kirkham, senior lecturer in biostatistics2
  1. 1Cochrane, St Albans House, London SW1Y 4QX, UK
  2. 2Department of Biostatistics, University of Liverpool, Liverpool, UK
  1. kdwan{at}cochrane.org

The BMJ published an investigation into the prevalence and impact of outcome reporting bias in randomised controlled trials included in Cochrane systematic reviews in 2010.1 It concluded that studies should not be excluded from reviews on the basis that there was “no relevant outcome data” (NROD) as failure to report review outcomes does not mean that the outcomes were not measured. Moreover, this recommendation is an expected methodological standard for Cochrane intervention reviews.2 Quality assurance screening of reviews by the Cochrane Editorial Unit has found that reviews still exclude studies on the basis of NROD. We investigated the proportion of Cochrane reviews excluding studies on the basis of NROD and whether the proportion had changed after review screening was implemented.

Methods

We included reviews from all Cochrane review groups published from June to August in 2013 (prescreening), 2014-15 (screening of all new reviews), and 2016 (screening on a referral basis by the Cochrane review groups). For each included review, we extracted the number of included studies, the number of excluded studies, and the number of studies excluded owing to NROD. To determine whether studies were excluded owing to NROD, we scrutinised the relevant sections of the review (methods, results, and characteristics of studies). We resolved any uncertainties about the reasons for exclusion through discussion. If a review excluded a study owing to NROD, we checked the review protocol to ascertain whether NROD was a prespecified criterion for study exclusion. We calculated the proportion of reviews excluding studies owing to NROD for each year as well as the relative risks (RR) and 95% confidence intervals (CI), to determine whether full screening or referred screening reduced the number of reviews excluding studies owing to NROD.

Results

We identified 434 new reviews that were published in the reference period. In the prescreening period, over a quarter of reviews excluded studies based on NROD, which fell to under a quarter in the new screening and referred screening phases (table 1).

Table 1

Reviews excluding studies owing to no relevant outcomes data (NROD)

View this table:

The results were almost significant for a reduced risk of reviews excluding studies owing to NROD if all new reviews were screened (RR 0.91, CI (0.81 to 1.03)) or were referred for screening (RR 0.93, CI (0.80 to 1.08)) compared with prescreening. Results were similar after removing reviews that prespecified exclusion owing to NROD.

Comment

The proportion of reviews excluding studies on the basis of NROD has fallen since introduction of the screening programme. But around a fifth of reviews are still excluding studies based on the lack of reporting of outcomes of interest in trial reports. If otherwise eligible studies are discarded based on failure to report outcome data, this constitutes research waste. Excluding outcome data from meta-analysis in this way can overestimate the treatment effect, which may lead to incorrect recommendations regarding treatment.1 Missing outcome data from excluded studies could be obtained from the trial authors or trial registries. Methods exist to help authors identify whether outcomes are likely to have been measured,13 and sensitivity analyses can assess whether the exclusion of data from studies is likely to affect the results.4 We need strategies to prevent authors publishing reviews with NROD as a reason for exclusion (such as specific checks at an earlier point in the process), and reasons for exclusion must be improved.

Footnotes

  • Competing interests: KMD is statistical editor for Cochrane. JJK and PRW have no competing interests.

References

View Abstract