Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological studyBMJ 2020; 368 doi: https://doi.org/10.1136/bmj.l6802 (Published 21 January 2020) Cite this as: BMJ 2020;368:l6802
- Helene Moustgaard, physician14,
- Gemma L Clayton, senior research associate in epidemiology5,
- Hayley E Jones, senior lecturer in medical statistics and data science5,
- Isabelle Boutron, professor of epidemiology6,
- Lars Jørgensen, physician4,
- David R T Laursen, doctoral student14,
- Mette F Olsen, postdoctoral fellow4,
- Asger Paludan-Müller, doctoral student4,
- Philippe Ravaud, professor of epidemiology6,
- Jelena Savović, senior research fellow5 7,
- Jonathan A C Sterne, professor of statistics and epidemiology5 7 8,
- Julian P T Higgins, professor of evidence synthesis5 7,
- Asbjørn Hróbjartsson, professor of evidence-based medicine13
- 1Centre for Evidence-Based Medicine Odense (CEBMO), Odense University Hospital, Kløvervænget 10, DK-5000 Odense C, Denmark
- 2Open Patient data Explorative Network (OPEN), Odense University Hospital, Odense, Denmark
- 3Department of Clinical Research, University of Southern Denmark, Odense, Denmark
- 4Nordic Cochrane Centre, Copenhagen, Denmark
- 5Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
- 6Cochrane France, Hôpital Hôtel-Dieu, Paris, France
- 7The National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West) at University Hospitals Bristol NHS Foundation Trust, Bristol, UK
- 8NIHR Bristol Biomedical Research Centre, University of Bristol, Bristol, UK
- Correspondence to: H Moustgaard @HeleneMoustgaa1 on Twitter) (or
- Accepted 19 November 2019
Objectives To study the impact of blinding on estimated treatment effects, and their variation between trials; differentiating between blinding of patients, healthcare providers, and observers; detection bias and performance bias; and types of outcome (the MetaBLIND study).
Design Meta-epidemiological study.
Data source Cochrane Database of Systematic Reviews (2013-14).
Eligibility criteria for selecting studies Meta-analyses with both blinded and non-blinded trials on any topic.
Review methods Blinding status was retrieved from trial publications and authors, and results retrieved automatically from the Cochrane Database of Systematic Reviews. Bayesian hierarchical models estimated the average ratio of odds ratios (ROR), and estimated the increases in heterogeneity between trials, for non-blinded trials (or of unclear status) versus blinded trials. Secondary analyses adjusted for adequacy of concealment of allocation, attrition, and trial size, and explored the association between outcome subjectivity (high, moderate, low) and average bias. An ROR lower than 1 indicated exaggerated effect estimates in trials without blinding.
Results The study included 142 meta-analyses (1153 trials). The ROR for lack of blinding of patients was 0.91 (95% credible interval 0.61 to 1.34) in 18 meta-analyses with patient reported outcomes, and 0.98 (0.69 to 1.39) in 14 meta-analyses with outcomes reported by blinded observers. The ROR for lack of blinding of healthcare providers was 1.01 (0.84 to 1.19) in 29 meta-analyses with healthcare provider decision outcomes (eg, readmissions), and 0.97 (0.64 to 1.45) in 13 meta-analyses with outcomes reported by blinded patients or observers. The ROR for lack of blinding of observers was 1.01 (0.86 to 1.18) in 46 meta-analyses with subjective observer reported outcomes, with no clear impact of degree of subjectivity. Information was insufficient to determine whether lack of blinding was associated with increased heterogeneity between trials. The ROR for trials not reported as double blind versus those that were double blind was 1.02 (0.90 to 1.13) in 74 meta-analyses.
Conclusion No evidence was found for an average difference in estimated treatment effect between trials with and without blinded patients, healthcare providers, or outcome assessors. These results could reflect that blinding is less important than often believed or meta-epidemiological study limitations, such as residual confounding or imprecision. At this stage, replication of this study is suggested and blinding should remain a methodological safeguard in trials.
Contributors: AH and HM conceived and organised the study, interpreted the results, and drafted the manuscript. HM also extracted data. GLC analysed the data, interpreted results, and drafted the manuscript. HEJ, JS, IB, PR, JPTH, and JACS conceived the study, interpreted the results, and drafted the manuscript. LJ, DRTL, AP-M, and MFO extracted data and drafted the manuscript. HM is guarantor. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
Funding: This study received no specific funding. GLC was funded by a PhD studentship from the Medical Research Council (MRC) Hubs for Trials Methodology Research. HEJ was supported by an MRC Career Development Award in Biostatistics (MR/M014533/1). JS and JPTH are supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care West (CLAHRC West). JACS and JPTH are NIHR senior investigators (NF-SI-0611-10168 and NF-SI-0617-10145, respectively), are supported by NIHR Bristol Biomedical Research Centre at University Hospitals Bristol NHS Foundation Trust and the University of Bristol, and are members of the MRC Integrative Epidemiology Unit at the University of Bristol. The views expressed are those of the author(s) and not necessarily those of the National Health Service, the NIHR, or the UK Department of Health and Social Care.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not required.
Data sharing: Dataset available from the corresponding author after a post-publication period of 1 year allowing time for follow-up projects.
The lead author affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.
Dissemination to participants and related patient and public communities: We plan to present our findings at national and international scientific meetings. We also plan to use social media outlets to disseminate findings. We will consider the implication of our findings for assessing the risk of bias in results of randomised trials using version 2 of the Cochrane risk of bias assessment tool.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.