Re: The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews
To the Editor: The prevalence and impact of outcome reporting bias in randomized controlled trials (RCTs) within Cochrane systematic reviews has previously been investigated [1]. A recommendation from this research was that studies should not be excluded from reviews on the basis that there was ‘no relevant outcome data’ (NROD), as failure to report on review outcomes does not imply that the outcomes were not measured. Moreover, this recommendation is an expected methodological standard for Cochrane intervention reviews [2]. Quality assurance screening of reviews carried out by the Cochrane Editorial Unit (CEU) has identified that reviews still exclude studies on the basis of NROD. We investigated the proportion of Cochrane reviews excluding studies on the basis of NROD and whether the proportion had changed following the implementation of review screening.
Methods. New Cochrane reviews were included from all Cochrane review groups published from June to August in 2013 (pre-screening), 2014 (screening of all new reviews), 2015 (screening of all new reviews) and 2016 (screening based on a referral basis by the Cochrane review groups). For each included review, investigators extracted the number of included studies, the number of excluded studies and the number of excluded studies due to NROD. To determine whether studies were excluded due to NROD, the relevant methods, results and characteristics of studies section of the review were scrutinised. Any uncertainties regarding the reasons for excluded studies were resolved through discussion between the investigators. If a review excluded a study due to NROD, the review protocol was checked to ascertain whether exclusion based on NROD was a pre-specified criterion for study exclusion. The proportion of reviews excluding studies due to NROD for each year was calculated. Relative risks (RR) and 95% confidence intervals (CI) were calculated to determine whether full screening or referred screening reduced the number of reviews excluding studies due to NROD.
Results. 434 new reviews were identified in the reference period. Over a quarter of reviews excluded studies based on NROD in the pre-screening period, while this figure reduced to under a quarter in the new review screen and referred screening phases (TABLE See http://www.outcome-reporting-bias.org/Uploads/ExcludedStudies.pdf). The result was almost significant for a reduced risk of reviews excluding studies due to NROD if all new reviews were screened (RR 0.91 CI (0.81, 1.03)) or were referred for screening (RR 0.93 CI (0.80, 1.08)) compared to pre-screening. Results were similar when removing reviews that pre-specified that studies would be excluded due to NROD.
Comment. Since the CEU introduced the screening programme the percentage of reviews excluding studies on the basis of NROD has reduced. However, around a fifth of reviews are still excluding studies based on the lack of reporting of outcomes of interest in trial reports. Restricting synthesis to only studies that report on relevant outcome constitutes research waste, if other, otherwise eligible studies are discarded based on failure to report outcome data. Excluding outcome data from meta-analysis in this way has previously been shown to overestimate the treatment effect, which may potentially lead to incorrect recommendations regarding treatment [1]. Potential missing outcome data from excluded studies could be obtained from contact with trial authors or results posted on trial registries. Methods are available to help authors identify whether outcomes are likely to have been measured [1, 3] and sensitivity analyses have been developed to assess whether the exclusion of data from studies is likely to impact on the results [4]. Future strategies are needed (e.g. specific checks at an earlier point in the process) to prevent authors publishing reviews with NROD as a reason for exclusion and reasons for exclusions need to be improved.
References
1. Kirkham, J.J., et al., The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ, 2010. 340: p. c365.
2. Higgins, J.P.T., et al., Methodological Expectations of Cochrane Intervention Reviews. 2016, Cochrane: London.
3. Saini, P., et al., Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ, 2014. 349: p. g6501.
4. Copas, J., et al., A model-based correction for outcome reporting bias in meta-analysis. Biostatistics, 2014. 15(2): p. 370-83.
Competing interests:
KMD is Statistical Editor for Cochrane. JJK and PRW have no competing interests.
14 August 2017
Kerry M Dwan
Statistical Editor
Paula R Williamson and Jamie J Kirkham
Cochrane
St Albans House, 57-59 Haymarket, London SW1Y 4QX, UK
Rapid Response:
Re: The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews
To the Editor: The prevalence and impact of outcome reporting bias in randomized controlled trials (RCTs) within Cochrane systematic reviews has previously been investigated [1]. A recommendation from this research was that studies should not be excluded from reviews on the basis that there was ‘no relevant outcome data’ (NROD), as failure to report on review outcomes does not imply that the outcomes were not measured. Moreover, this recommendation is an expected methodological standard for Cochrane intervention reviews [2]. Quality assurance screening of reviews carried out by the Cochrane Editorial Unit (CEU) has identified that reviews still exclude studies on the basis of NROD. We investigated the proportion of Cochrane reviews excluding studies on the basis of NROD and whether the proportion had changed following the implementation of review screening.
Methods. New Cochrane reviews were included from all Cochrane review groups published from June to August in 2013 (pre-screening), 2014 (screening of all new reviews), 2015 (screening of all new reviews) and 2016 (screening based on a referral basis by the Cochrane review groups). For each included review, investigators extracted the number of included studies, the number of excluded studies and the number of excluded studies due to NROD. To determine whether studies were excluded due to NROD, the relevant methods, results and characteristics of studies section of the review were scrutinised. Any uncertainties regarding the reasons for excluded studies were resolved through discussion between the investigators. If a review excluded a study due to NROD, the review protocol was checked to ascertain whether exclusion based on NROD was a pre-specified criterion for study exclusion. The proportion of reviews excluding studies due to NROD for each year was calculated. Relative risks (RR) and 95% confidence intervals (CI) were calculated to determine whether full screening or referred screening reduced the number of reviews excluding studies due to NROD.
Results. 434 new reviews were identified in the reference period. Over a quarter of reviews excluded studies based on NROD in the pre-screening period, while this figure reduced to under a quarter in the new review screen and referred screening phases (TABLE See http://www.outcome-reporting-bias.org/Uploads/ExcludedStudies.pdf). The result was almost significant for a reduced risk of reviews excluding studies due to NROD if all new reviews were screened (RR 0.91 CI (0.81, 1.03)) or were referred for screening (RR 0.93 CI (0.80, 1.08)) compared to pre-screening. Results were similar when removing reviews that pre-specified that studies would be excluded due to NROD.
Comment. Since the CEU introduced the screening programme the percentage of reviews excluding studies on the basis of NROD has reduced. However, around a fifth of reviews are still excluding studies based on the lack of reporting of outcomes of interest in trial reports. Restricting synthesis to only studies that report on relevant outcome constitutes research waste, if other, otherwise eligible studies are discarded based on failure to report outcome data. Excluding outcome data from meta-analysis in this way has previously been shown to overestimate the treatment effect, which may potentially lead to incorrect recommendations regarding treatment [1]. Potential missing outcome data from excluded studies could be obtained from contact with trial authors or results posted on trial registries. Methods are available to help authors identify whether outcomes are likely to have been measured [1, 3] and sensitivity analyses have been developed to assess whether the exclusion of data from studies is likely to impact on the results [4]. Future strategies are needed (e.g. specific checks at an earlier point in the process) to prevent authors publishing reviews with NROD as a reason for exclusion and reasons for exclusions need to be improved.
References
1. Kirkham, J.J., et al., The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ, 2010. 340: p. c365.
2. Higgins, J.P.T., et al., Methodological Expectations of Cochrane Intervention Reviews. 2016, Cochrane: London.
3. Saini, P., et al., Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ, 2014. 349: p. g6501.
4. Copas, J., et al., A model-based correction for outcome reporting bias in meta-analysis. Biostatistics, 2014. 15(2): p. 370-83.
Competing interests: KMD is Statistical Editor for Cochrane. JJK and PRW have no competing interests.