BMJ  2004;328:432 (21 February), doi:10.1136/bmj.37952.631667.EE (published 22 January 2004)

Paper

Turning a blind eye: the success of blinding reported in a random sample of randomised, placebo controlled trials

¹ Ottawa Health Research Institute, Clinical Epidemiology Program, 501 Smyth Road, Box 201, Ottawa, ON, Canada KIH 8L6, ² Departments of Human Genetics & Pediatrics and Biomedical Ethics Unit, McGill University, Montreal, QC, Canada, ³ Department of Epidemiology and Biostatistics, McGill University, ⁴ Research Ethics and Regulation Group, Faculty of Law, University of Toronto, Toronto, ON, Canada

Abstract

Objective To examine the reporting and success of double blinding in a sample of randomised, placebo controlled trials from leading general medicine and psychiatry journals.

Methods Identification of placebo controlled, randomised controlled trials from prespecified general medical and psychiatric journals indexed on Medline between 1 January 1998 and 1 October 2001, from which a random sample of 200 randomised clinical trials was chosen, of which 191 trials were evaluated.

Results Only seven of the 97 (7%) general medicine trials provided evidence on the success of blinding, with five reporting that the success of blinding was imperfect. In trials from psychiatric journals, the success of blinding was reported in eight of the 94 trials, with four reporting that the blinding was imperfect. Overall, only four of the 191 (2%) trials assessed blinding in the participants and either the outcome assessors or the investigators.

Conclusions The current lack of reporting on the success of blinding provides little evidence that success of blinding is maintained in placebo controlled trials. Trialists and editors should make a concerted effort to incorporate, report, and publish such information and its potential effect on study results.

Introduction

Although the definition of double blind varies,¹ we consider a trial to be double blind when the patient, investigators, and outcome assessors are unaware of the patient's assigned treatment throughout the conduct of the trial.² Placebos are commonly used as an inactive treatment to achieve double blinding, especially when no existing effective treatment is available. Sometimes, placebos are proposed instead of a standard existing treatment or standard care to ensure assay sensitivity. This is the ability of a trial to distinguish effective interventions from ineffective interventions. It depends on the effect size that is to be detected, so the investigators need to know the anticipated effects of the control intervention. Placebos are ideal as their anticipated benefits are known to be marginal. If the blinding of the placebo arm is not effective then the protection against expectation effects, biased assessment, contamination, and co-intervention are all lost. Because of the importance of the success of blinding, the Consolidated Standards for Reporting of Trials (CONSORT) Group has explicitly incorporated the issue. Section 11(b) of the CONSORT statement states that the success of blinding is to be reported in the publication.³

It is not sufficient that trials describe themselves as double blind. It is also important that the efficacy of the blinding is actually assessed, and an assessment of the face validity of the double blinding is needed. To assess the reporting and success of double blinding, we chose a random sample of randomised, placebo controlled trials from leading journals in general medicine and psychiatry. Although we have focussed on placebo controlled trials, the issues discussed also arise in double blind trials with active controls.

Methods

We selected five general medical journals (JAMA, New England Journal of Medicine, BMJ, Lancet, and Annals of Internal Medicine) and four journals in psychiatry (Archives of General Psychiatry, Journal of Clinical Psychiatry, British Journal of Psychiatry, and American Journal of Psychiatry). Our Medline search used publication type "randomised controlled trial" and the MeSH term "placebo-controlled" to identify placebo controlled randomised trials that were indexed on Medline between the dates of 1 January 1998 and 1 October 2001 and published in these nine journals. We randomly selected 100 trials from the general medicine journals and 100 trials from the psychiatry journals. We reasoned that 100 trials from each discipline was a manageable number to abstract and an adequate number to obtain a good estimate of the number of trials reporting the success of blinding, and we performed no formal sample size calculations.

Data were gathered from the trials using abstraction forms. A trial indicating that a "similar" placebo was used, but not specifying how it was similar, was scored as "not mentioned." Six people abstracted all the data. At least two people independently abstracted data for each study. Either consensus or a third party resolved any differences.

Results

From a total of 473 randomised controlled trials in the general medicine literature and 192 trials in the psychiatric literature, we randomly chose 100 trials from each. Nine trials were not placebo controlled trials, leaving 97 trials from the general medicine literature and 94 trials from psychiatry literature for analysis.

General medicine
The table provides information on the type of interventions and placebos used in the 97 trials in general medicine.

View this table: [in this window] [in a new window]   Type of intervention and placebo. Values are number of trials

 

The matching characteristics of placebo to the intervention was reported in 51 (53%) of the trials; one trial (1%) also reported the dissimilarity between placebo and intervention. Appearance was the characteristic most often reported by investigators (46 of 51 trials), followed by taste (9 of 51 trials).

What is already known on this topic Placebo controls are commonly used in randomised trials to blind investigators, outcome assessors, and patients to treatment assignment Placebo controls have been advocated instead of existing effective treatment because they ensure assay sensitivity Unsuccessful double blinding results in a differential bias of effect measures What this study adds The success of blinding is not well reported The success of blinding in trials that do report is often poor Little evidence exists that placebos provide assay sensitivity

Only seven of the 97 trials (7%) provided evidence on the success of blinding (see bmj.com). All seven trials presented a method for assessing blinding. and all assessed the success of blinding in study participants. One trial also assessed the success of blinding in individuals assessing study outcome. Five of the trials presented blinding data for each trial arm, and reported that the success of blinding was imperfect. One trial presented overall aggregated data only, but did not comment qualitatively, or provide statistical tests of success of blinding. One trial provided no data and described blinding as successful without further comment.

Psychiatry
Most psychiatry trials used pharmacological interventions. The matching characteristics between intervention and placebo were reported in 30 (32%) of trials, with appearance being the most often reported. Eight of the 94 trials reported evidence on successful blinding (see bmj.com). Of these, six assessed the success of blinding in patients. Two studies provided blinding data for both subjects and outcome assessors; one study reported blinding data for both treatment administrators and outcome assessors; and one study provided data for treatment administrators only. Six of the eight studies presented a method for blinding assessment in the Methods or Results section of the article and the other two presented it in the Discussion section. Four of the trials presented blinding data broken down by treatment allocation; one trial presented aggregated data and did not provide data broken down by treatment allocation; and two presented no data on blinding. Of the eight trials, the blinding was reported as less than optimal in four.

Discussion

Our examination of the success of blinding challenges the notion that placebo controlled trials inherently possess assay sensitivity. There is a failure among investigators and journals in reporting the success of blinding. Only 15 of the 191 trials (8%) provided such information, be it qualitative or quantitative. Of the 15 trials, only five trials reported that blinding was successful, and of these, three did not present any quantitative data analysis to support their claim.

Only four trials assessed blinding in both the participants and either the outcome assessors or the investigators. Thus, the face validity of the double blinding was only reported in four of the 191 articles (2%). Furthermore, the quality of evidence in the few studies that reported on the success of blinding is weak on two fronts: the quality of the data and the evidence that blinding was successful.

We would like to see item 11(b) of CONSORT revised to require the assessment of blinding for all double blind randomised trials. Trialists have an ethical responsibility to justify the use of a placebo for blinding purposes in their research protocol and informed consent procedures. Thus, it seems reasonable to suggest that an assessment of the success of blinding is necessary. If blinding is not assessed, we may delude ourselves as to exactly what information we gain from incorporating a placebo comparison. The types of trials that will particularly benefit are trials with subjective outcomes or outcomes reported by patients (for example, quality of life instruments), or trials where the side effects are well known. The lack of successful blinding can bias observed estimates of effect. Although this bias is differential, its direction may not be easily ascertained.

We believe that trialists need to report a minimum set of information. This includes the counts of all patients allocated to each treatment; the counts of patients who guess treatment assignment by the group to which they were allocated; the counts of correct guesses and of those who are undecided; the analytical methods and results used to assess success of blinding; and the author's interpretation of the efficacy of blinding and the effect on study results.

Trialists and editors need to make a concerted effort to incorporate, report, and publish information about the success of blinding and its potential effect on study results. We need evidence before we can assert that assay sensitivity exists in randomised, double blind, placebo controlled trials.

---

This is the abridged version of an article that was first posted on bmj.com on 22 January 2004: http://bmj.com/cgi/doi/10.1136/bmj.37952.631667.EE

We thank Julie Comber and Jennifer Marshall for article retrieval and data collection.

Contributors: See bmj.com

Funding: This work was funded in part by the Canadian Institutes of Health Research.

Competing interests: None declared.

Ethical approval: Not required.

References

1. Devereaux PJ, Manns BJ, Ghali WA, Quan H, Lacchetti C, Montori VM, et al. Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. JAMA 2001;285: 2000-3.[Abstract/Free Full Text]
2. Schulz KF, Chalmers I, Altman DG. The landscape and lexicon of blinding in randomized trials. Ann Intern Med. 2002;136: 254-9.[Free Full Text]
3. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001;134: 663-94.[Abstract/Free Full Text]
4. Sackeim HA, Haskett RF, Mulsant BH, Thase ME, Mann JJ, Pettinati HM, et al. Continuation pharmacotherapy in the prevention of relapse following electroconvulsive therapy: a randomized controlled trial. JAMA 2001;285: 1299-307.[Abstract/Free Full Text]

(Accepted 11 November 2003)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Articles

Future of psychotherapy in the NHS: Control groups play important part

D B Double
BMJ 2004 329: 514. [Extract] [Full Text]

Turning a blind eye: Testing the success of blinding and the CONSORT statement

Douglas G Altman, Kenneth F Schulz, and David Moher
BMJ 2004 328: 1135. [Extract] [Full Text]

Turning a blind eye: Authors have blinkered view of blinding

Stephen J Senn
BMJ 2004 328: 1135-1136. [Extract] [Full Text]

Turning a blind eye: Why we don't test for blindness at the end of our trials

David L Sackett
BMJ 2004 328: 1136. [Extract] [Full Text]

Evidence of success of blinding in clinical trials is scant
BMJ 2004 328: 0. [Full Text]

This article has been cited by other articles:

• Howick, J., Glasziou, P., Aronson, J. K (2009). The evolution of evidence hierarchies: what can Bradford Hill's 'guidelines for causation' contribute?. JRSM 102: 186-194 [Full text]  
• Filion, K. B. MSc, Belisle, P. MSc, Joseph, L. PhD, Paradis, G. MD MSc, Rinfret, S. MD MSc, Eisenberg, M. J. MD MPH (2008). Smoking cessation trials. CMAJ 179: 1038-1038 [Full text]  
• Sackett, D. L (2007). Commentary: Measuring the success of blinding in RCTs: don't, must, can't or needn't?. Int J Epidemiol 36: 664-665 [Full text]  
• Hrobjartsson, A, Forfang, E, Haahr, M., Als-Nielsen, B, Brorson, S (2007). Blinded trials taken to the test: an analysis of randomized clinical trials that report tests for the success of blinding. Int J Epidemiol 36: 654-663 [Abstract] [Full text]  
• Poolman, R. W., Struijs, P. A.A., Krips, R., Sierevelt, I. N., Marti, R. K., Farrokhyar, F., Bhandari, M. (2007). Reporting of Outcomes in Orthopaedic Randomized Trials: Does Blinding of Outcome Assessors Matter?. JBJS 89: 550-558 [Abstract] [Full text]  
• Vickrey, B. G., Mittman, B. S., Connor, K. I., Pearson, M. L., Della Penna, R. D., Ganiats, T. G., DeMonte, R. W. Jr., Chodosh, J., Cui, X., Vassar, S., Duan, N., Lee, M. (2006). The Effect of a Disease Management Intervention on Quality and Outcomes of Dementia Care: A Randomized, Controlled Trial.. ANN INTERN MED 145: 713-726 [Abstract] [Full text]  
• Bausell, R. B., Lao, L., Bergman, S., Lee, W.-L., Berman, B. M. (2005). Is Acupuncture Analgesia an Expectancy Effect?: Preliminary Evidence Based on Participants' Perceived Assignments in Two Placebo-Controlled Trials. Eval Health Prof 28: 9-26 [Abstract]  
• Matthews, S B, Waud, J P, Roberts, A G, Campbell, A K (2005). Systemic lactose intolerance: a new perspective on an old problem. Postgrad. Med. J. 81: 167-173 [Abstract] [Full text]  
• Double, D B (2004). Future of psychotherapy in the NHS: Control groups play important part. BMJ 329: 514-514 [Full text]  
• Altman, D. G, Schulz, K. F, Moher, D. (2004). Turning a blind eye: Testing the success of blinding and the CONSORT statement. BMJ 328: 1135-1135 [Full text]  
• Senn, S. J (2004). Turning a blind eye: Authors have blinkered view of blinding. BMJ 328: 1135-1136 [Full text]  
• Sackett, D. L (2004). Turning a blind eye: Why we don't test for blindness at the end of our trials. BMJ 328: 1136-1136 [Full text]  

Rapid Responses:

Read all Rapid Responses

A Blinkered view of Blinding

Stephen J Senn
bmj.com, 20 Feb 2004 [Full text]

Why we don't test for blindness at the end of our trials.

David L. Sackett
bmj.com, 20 Feb 2004 [Full text]

Changing the mindset about unblinding in clinical trials

D B Double
bmj.com, 21 Feb 2004 [Full text]

Testing the success of blinding and the CONSORT Statement

Douglas G. Altman, et al.
bmj.com, 21 Feb 2004 [Full text]

The Blind leading the Blind?

Michael Goodyear
bmj.com, 24 Feb 2004 [Full text]

Widening the field of vision

Stan Shapiro, et al.
bmj.com, 24 Feb 2004 [Full text]

Turning a blind eye to bias in clinical trials

D B Double
bmj.com, 26 Feb 2004 [Full text]

Blind evaluation of blinding process

Santosh K Chaturvedi
bmj.com, 2 Mar 2004 [Full text]

Failure to assess blinding can make clinical trials hard to interpret

Andrei SP Brennan, et al.
bmj.com, 23 Mar 2004 [Full text]

The Paradox of the Placebo

Raywat S. Deonandan, et al.
bmj.com, 16 Jun 2004 [Full text]

Placebo-controlled: is it really a MeSH term?

Jafar Kolahi
bmj.com, 29 Sep 2006 [Full text]

Online poll

Find out more>>