Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessorsBMJ 2012; 344 doi: https://doi.org/10.1136/bmj.e1119 (Published 27 February 2012) Cite this as: BMJ 2012;344:e1119
- Asbjørn Hróbjartsson, senior researcher1,
- Ann Sofia Skou Thomsen, research associate1,
- Frida Emanuelsson, research associate1,
- Britta Tendal, postdoctoral fellow1,
- Jørgen Hilden, associate professor of biostatistics2,
- Isabelle Boutron, associate professor of epidemiology 3,
- Philippe Ravaud, professor of epidemiology3,
- Stig Brorson, orthopaedic surgeon4
- 1Nordic Cochrane Centre, Rigshospitalet Department 3343, Blegdamsvej 9, 2100 Copenhagen Ø, Denmark
- 2Department of Biostatistics, University of Copenhagen, Copenhagen
- 3French Cochrane Centre, Assistance Publique (Hotel Dieu), INSERM U738, Université Paris Descartes, France
- 4Department of Orthopaedic Surgery, Herlev University Hospital, Copenhagen
- Correspondence to: A Hróbjartsson
- Accepted 22 December 2011
Objective To evaluate the impact of non-blinded outcome assessment on estimated treatment effects in randomised clinical trials with binary outcomes.
Design Systematic review of trials with both blinded and non-blinded assessment of the same binary outcome. For each trial we calculated the ratio of the odds ratios—the odds ratio from non-blinded assessments relative to the corresponding odds ratio from blinded assessments. A ratio of odds ratios <1 indicated that non-blinded assessors generated more optimistic effect estimates than blinded assessors. We pooled the individual ratios of odds ratios with inverse variance random effects meta-analysis and explored reasons for variation in ratios of odds ratios with meta-regression. We also analysed rates of agreement between blinded and non-blinded assessors and calculated the number of patients needed to be reclassified to neutralise any bias.
Data Sources PubMed, Embase, PsycINFO, CINAHL, Cochrane Central Register of Controlled Trials, HighWire Press, and Google Scholar.
Eligibility criteria for selecting studies Randomised clinical trials with blinded and non-blinded assessment of the same binary outcome.
Results We included 21 trials in the main analysis (with 4391 patients); eight trials provided individual patient data. Outcomes in most trials were subjective—for example, qualitative assessment of the patient’s function. The ratio of the odds ratios ranged from 0.02 to 14.4. The pooled ratio of odds ratios was 0.64 (95% confidence interval 0.43 to 0.96), indicating an average exaggeration of the non-blinded odds ratio by 36%. We found no significant association between low ratios of odds ratios and scores for outcome subjectivity (P=0.27); non-blinded assessor’s overall involvement in the trial (P=0.60); or outcome vulnerability to non-blinded patients (P=0.52). Blinded and non-blinded assessors agreed in a median of 78% of assessments (interquartile range 64-90%) in the 12 trials with available data. The exaggeration of treatment effects associated with non-blinded assessors was induced by the misclassification of a median of 3% of the assessed patients per trial (1-7%).
Conclusions On average, non-blinded assessors of subjective binary outcomes generated substantially biased effect estimates in randomised clinical trials, exaggerating odds ratios by 36%. This bias was compatible with a high rate of agreement between blinded and non-blinded outcome assessors and driven by the misclassification of few patients.
We thank the following researchers for sharing unpublished outcome data with us: Andrew Jull and the Clinical Trials Research Unit (CTRU) at the University of Auckland, Alexandre Valentin-Opran, Nicky Cullum, Tim Reynolds, Peggy Vandervoort, and George C Ebers (individual patient data); and Daniel Burkhoff, Amy P Murtha, and Cheryl Iglesia (detailed outcome data). We also thank Peter C Gøtzsche for valuable comments on previous versions of the manuscript.
Contributors: AH conceived the idea and design, organised the study, and wrote the first draft of the manuscript. ASST and AH developed the search strategy. ASST, FE, BT, SB, and AH did the non-masked data collection. IB and PR did the masked data collection, supplemented by SB, BT, and AH. AH and JH did the statistical analyses. All authors discussed the result and commented on the manuscript. AH is guarantor.
Funding: The study was partially funded by the Danish Council for Independent Research: Medical sciences. The funder had no influence on study design and the collection, analysis, and interpretation of data and the writing of the article and the decision to submit it for publication.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not required.
Data sharing: Statistical code and dataset available from the corresponding author.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.