BMJ  2004;328:432 (21 February), doi:10.1136/bmj.37952.631667.EE (published 22 January 2004)

Paper

Turning a blind eye: the success of blinding reported in a random sample of randomised, placebo controlled trials

¹ Ottawa Health Research Institute, Clinical Epidemiology Program, 501 Smyth Road, Box 201, Ottawa, ON, Canada K1H 8L6, ² Departments of Human Genetics & Pediatrics and Biomedical Ethics Unit, McGill University, Montreal, QC, Canada, ³ Department of Epidemiology and Biostatistics, McGill University, ⁴ Research Ethics and Regulation Group, Faculty of Law, University of Toronto, Toronto, ON, Canada

Abstract

Objective To examine the reporting and success of double blinding in a sample of randomised, placebo controlled trials from leading general medicine and psychiatry journals.

Methods Identification of placebo controlled, randomised controlled trials from prespecified general medical and psychiatric journals indexed on Medline between 1 January 1998 and 1 October 2001, from which a random sample of 200 randomised clinical trials was chosen, of which 191 trials were evaluated.

Results Only seven of the 97 (7%) general medicine trials provided evidence on the success of blinding, with five reporting that the success of blinding was imperfect. In trials from psychiatric journals, the success of blinding was reported in eight of the 94 trials, with four reporting that the blinding was imperfect. Overall, only four of the 191 (2%) trials assessed blinding in the participants and either the outcome assessors or the investigators.

Conclusions The current lack of reporting on the success of blinding provides little evidence that success of blinding is maintained in placebo controlled trials. Trialists and editors should make a concerted effort to incorporate, report, and publish such information and its potential effect on study results.

Introduction

Although the definition of double blind varies,¹ we consider a trial to be double blind when the patient, investigators, and outcome assessors are unaware of the patient's assigned treatment throughout the conduct of the trial.² Placebos are commonly used as an inactive treatment to achieve double blinding. Active placebos, with which symptoms or side effects are imitated, can also be used. Placebos are justly used when no existing effective treatment is available. Sometimes, placebos are proposed instead of a standard existing treatment or standard care to ensure assay sensitivity. That is, to demonstrate the effectiveness of a new treatment, it must be demonstrated against a "clean" control. The argument is that although the new treatment may be found to be as effective or more effective than standard treatment in a clinical trial, both treatments may very well be ineffective. Assay sensitivity is the ability of a trial to distinguish effective interventions from ineffective interventions. It depends on the effect size that is to be detected. As such, the investigators need to know the anticipated effects of the control intervention. It is argued that placebos are the ideal choice as their anticipated benefits are known to be marginal. This argument is predicated on the belief that participants, investigators, and outcome assessors remain blinded to the treatment assignment. If the blinding of the placebo arm is not effective then the protection against expectation effects, biased assessment, contamination, and co-intervention are all lost. The observed superiority of a new treatment over placebo could merely be a consequence of loss of this control—and an ineffective new treatment would spuriously seem to be superior. Because of the importance of the success of blinding, the Consolidated Standards for Reporting of Trials (CONSORT) Group has explicitly incorporated the issue. Section 11(b) of the CONSORT statement states that the success of blinding is to be reported in the publication.³

It is not sufficient that trials describe themselves as double blind. It is also important that the efficacy of the blinding is actually assessed. In other words, an assessment of the face validity of the double blinding is needed. To assess the reporting and success of double blinding, we chose a random sample of randomised, placebo controlled trials from leading journals in general medicine and psychiatry. Although we have focused on placebo controlled trials, the issues discussed also arise in double blind trials with active controls.

Methods

For this study we selected five of the top general medical journals (JAMA, New England Journal of Medicine, BMJ, Lancet, and Annals of Internal Medicine) and four of the top journals in psychiatry (Archives of General Psychiatry, Journal of Clinical Psychiatry, British Journal of Psychiatry, and American Journal of Psychiatry). Our Medline search used publication type "randomised controlled trial" and the MeSH term "placebo-controlled" to identify placebo controlled randomised trials that were indexed on Medline between the dates of 1 January 1998 and 1 October 2001 and published in these nine journals. All citations from this search were then consecutively numbered, and a random number generator was used to select 100 trials from the general medicine journals and 100 trials from the psychiatry journals. We reasoned that 100 trials from each discipline was a manageable number to abstract and an adequate number to obtain a good estimate of the number of trials reporting the success of blinding, and we performed no formal sample size calculations.

Data abstraction forms were developed and included document identification, the type of interventions, type of placebo, matching characteristics of placebo to intervention, who was blinded, and the evidence of successful blinding. A trial indicating that a "similar" placebo was used but did not specify how it was similar was scored as "not mentioned" (our rationale is that the term "similar" is vague and therefore inadequate). The page number and location for each data item was also recorded. Six people abstracted all the data. At least two people independently abstracted data for each study. Either consensus or a third party resolved any differences. All data were entered into an electronic database (Microsoft Excel 2000).

Results

The Medline search identified a total of 473 randomised controlled trials in the general medicine literature and 192 trials in the psychiatric literature, from which we randomly chose 100 trials from general medicine and 100 trials from psychiatry. Nine trials were removed from further analysis as they were not placebo controlled trials despite being identified as such in the systematic literature search. Thus, we evaluated 97 trials from the general medicine literature and 94 trials from psychiatry literature.

General medicine
Table 1 provides information on the type of interventions and placebos used in the 97 trials in general medicine. Eighty three per cent of all interventions were pharmacological. Nutritional supplements (9%) were the second most frequent intervention used. Sixteen of the 97 trials did not report the type of placebo used. Of trials that reported the type of placebo, an injection (either subcutaneous or intramuscular) was the most common (23; 27%), followed by tablet (20; 24%) and capsule (18; 21%).

View this table: [in this window] [in a new window]   Table 1 Type of intervention and placebo. Values are number of trials

 

The matching characteristics of placebo to the intervention was reported in 51 (53%) of the trials; one trial (1%) also reported the dissimilarity between placebo and intervention (table 2). Appearance was the characteristic most often reported by investigators (46 of 51 trials), followed by taste (9 of 51 trials).

View this table: [in this window] [in a new window]   Table 2 Number of studies reporting matching of characteristics of placebo to intervention

 

Only seven of the 97 trials (7%) provided evidence on the success of blinding (table 3).^4-10 All seven trials assessed the success of blinding in study participants. One trial assessed the success of blinding in individuals assessing study outcome.⁶ All seven trials presented a method for assessing blinding. Five of the trials presented blinding data for each trial arm, one trial presented overall aggregated data only, and one trial provided no data. Five trials reported that the success of blinding was imperfect.^{4 6-8 10} The trial that did not present blinding data described blinding as successful without further comment.⁹ The trial that reported aggregated blinding data did not comment, qualitatively, or provide statistical tests of success of blinding.⁵

View this table: [in this window] [in a new window]   Table 3 Reporting of blinding assessment in 97 trials in general medicine

 

Psychiatry
Most psychiatry trials used pharmacological interventions (table 1). Over 40% of the trials did not report the type of placebo used. Of the placebos reported, 78% were either a capsule or tablet.

The matching characteristics between intervention and placebo were reported in 30 (32%) of trials, with appearance being the most often reported (table 2). Eight of the 94 trials reported evidence on successful blinding (table 4).^11-18 Of these, six assessed the success of blinding in patients.^{12 14-18} Two studies provided blinding data for both subjects and outcome assessors^{16 17}; one study reported blinding data for both treatment administrators and outcome assessors,¹¹ and one study provided data for treatment administrators only.¹³ Six of the eight studies presented a method for blinding assessment in the Methods or Results section of the article and the other two presented it in the Discussion section. Four of the trials presented blinding data broken down by treatment allocation^{12 14 15 17}; one trial presented aggregated data and did not provide data broken down by treatment allocation¹³; and two presented no data on blinding.^{11 16} Of the eight trials, the blinding was reported as less than optimal in four.^{11 14 15 18}

View this table: [in this window] [in a new window]   Table 4 Reporting of blinding assessment in 94 trials in psychiatry

 

Discussion

The quality of reporting in clinical trials has evolved. Over the years, trialists have been held more accountable and responsible for the quality of trial reporting. This evolution began with the need for reporting the numbers of patients screened, enrolled, randomised, and analysed,¹⁹ and progressed to the reporting of patient withdrawals and its importance for the analysis and interpretation of study results.²⁰ Building on this progress, there is a need for trialists and journals routinely to report the methods of blinding and the subsequent success of this blinding.²¹

Our examination of the success of blinding challenges the notion that placebo controlled trials inherently possess assay sensitivity. Clearly, there is a failure among investigators and journals in reporting the success of blinding. Only 15 of the 191 trials (8%) provided such information, be it qualitative or quantitative. Of the 15 trials, only five trials reported that blinding was successful,^{9 12 13 16 17} and of these, three did not present any quantitative data analysis to support their claim.^{9 13 16}

Only four trials assessed blinding in both the participants and either the outcome assessors or the investigators.^{6 12 16 17} Thus, the face validity of the double blinding was only reported in four of the 191 articles (2%). This deficiency in reporting translates into a paucity of evidence that a placebo ensures a "clean" control. Furthermore, the quality of evidence in the few studies that reported on the success of blinding is weak on two fronts: the quality of the data and the evidence that blinding was successful. The success of blinding was described as less than optimal in nine of the 14 trials that reported on blinding, and of the five trials that reported that blinding was maintained, only two provided data to support their claim.^{12 17} Unfortunately, when we examined the data and analysis provided by these two trials we found that their claim of success is debatable.

We would like to see Item 11b of CONSORT revised to require the assessment of blinding for all double blind randomised trials. Trialists have an ethical responsibility to justify the use of a placebo for blinding purposes in their research protocol and informed consent procedures. Thus, it seems reasonable to suggest that an assessment of the success of blinding is necessary. If blinding is not assessed, we may delude ourselves as to exactly what information we gain from incorporating a placebo comparison. Although all trials should assess blinding, the types of trials that will particularly benefit are trials with subjective outcomes or outcomes reported by patients (for example, quality of life instruments), or trials where the side effects are well known. Even though there may be problems with analysing and interpreting the results of success, this does not provide a rationale for not doing it. Clearly, the lack of successful blinding can bias observed estimates of effect. Although this bias is differential, its direction may not be easily ascertained. We might anticipate that evidence of unsuccessful blinding in a "double blind" active versus placebo trial would result in a positive bias and hence lead to an overestimate of the treatment effect. However, unblinded patients receiving placebo may seek other treatments, especially if there is established effective treatment available, and this makes the extent and even the direction of bias difficult to determine.

We believe that trialists need to report a minimum set of information. This includes the counts of all patients allocated to each treatment; the counts of patients who guess treatment assignment by the group to which they were allocated; the counts of correct guesses and those who are undecided; the analytical methods and results used to assess success of blinding; and the author's interpretation of the efficacy of blinding and the effect on study results. The data abstracted for this study show a substantial lack of reporting with respect to these minimum, essential items, as illustrated by the number of vacant fields in tables 3 and 4.

What is already known on this topic Placebo controls are commonly used in randomised trials to blind investigators, outcome assessors, and patients to treatment assignment Placebo controls have been advocated instead of existing effective treatment because they ensure assay sensitivity Unsuccessful double blinding results in a differential bias of effect measures What this study adds The success of blinding is not well reported The success of blinding in trials that do report is often poor Little evidence exists that placebos provide assay sensitivity

The current lack of reporting on the success of blinding provides little evidence that success of blinding is maintained in placebo controlled trials. Trialists and editors need to make a concerted effort to incorporate, report, and publish such information and its potential effect on study results. The efficacy of the blinding cannot be assumed on theoretical grounds. We need evidence before we can assert that assay sensitivity exists in randomised, double blind, placebo controlled trials.

Amendment
This is Version 2 of the paper. In this version, the references in tables 3 and 4 have been corrected. They now start with reference 4 and end with reference 18 [in the previous version they ran from reference 2 to reference 16].

---

We thank Julie Comber and Jennifer Marshall for article retrieval and data collection.

Contributors: DF, KG, DW, and SS conceived and designed the study. DF collected, managed, and analysed the data. All authors interpreted the data and wrote the paper. DF is the guarantor.

Funding: This work was funded in part by the Canadian Institutes of Health Research.

Competing interests: None declared.

Ethical approval: Not required.

References

1. Devereaux PJ, Manns BJ, Ghali WA, Quan H, Lacchetti C, Montori VM, et al. Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials. JAMA 2001;285: 2000-3.[Abstract/Free Full Text]
2. Schulz KF, Chalmers I, Altman DG. The landscape and lexicon of blinding in randomized trials. Ann Intern Med. 2002;136: 254-9.[Free Full Text]
3. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001;134: 663-94.[Abstract/Free Full Text]
4. Sackeim HA, Haskett RF, Mulsant BH, Thase ME, Mann JJ, Pettinati HM, et al. Continuation pharmacotherapy in the prevention of relapse following electroconvulsive therapy: a randomized controlled trial. JAMA 2001;285: 1299-307.[Abstract/Free Full Text]
5. Rowe PC, Calkins H, DeBusk K, McKenzie R, Anand R, Sharma G, et al. Fludrocortisone acetate to treat neurally mediated hypotension in chronic fatigue syndrome: a randomized controlled trial. JAMA 2001;285: 52-9.[Abstract/Free Full Text]
6. Apfel SC, Schwartz S, Adornato BT, Freeman R, Biton V, Rendell M, et al. Efficacy and safety of recombinant human nerve growth factor in patients with diabetic polyneuropathy: a randomized controlled trial. rhNGF Clinical Investigator Group. JAMA 2000;284: 2215-21.[Abstract/Free Full Text]
7. Von Schacky C, Angerer P, Kothny W, Theisen K, Mudra H. The effect of dietary omega-3 fatty acids on coronary atherosclerosis. A randomised, double-blind, placebo-controlled trial. Ann Intern Med 1999;130: 554-62.[Abstract/Free Full Text]
8. Sandler RS, Zorich NL, Filloon TG, Wiseman HB, Lietz DJ, Brock MH, et al. Gastrointestinal symptoms in 3181 volunteers ingesting snack foods containing olestra or triglycerides. A 6-week randomised, placebo-controlled trial. Ann Intern Med 1999;130: 253-61.[Abstract/Free Full Text]
9. Blondal T, Gudmundsson LJ, Olafsdottir I, Gustavsson G, Westin A. Nicotine nasal spray with nicotine patch for smoking cessation: randomised trial with six year follow up. BMJ 1999;318: 285-8.[Abstract/Free Full Text]
10. Shlay JC, Chaloner K, Max MB, Flaws B, Reichelderfer P, Wentworth D, et al. Acupuncture and amitriptyline for pain due to HIV-related peripheral neuropathy: a randomised controlled trial. Terry Beirn Community Programs for Clinical Research on AIDS. JAMA 1998;280: 1590-5.[Abstract/Free Full Text]
11. Wisner KL, Perel JM, Peindl KS, Hanusa BH, Findling RL, Rapport D. Prevention of recurrent postpartum depression: a randomised clinical trial. J Clin Psychiatry 2001;62: 82-6.[ISI][Medline]
12. Warner J, Metcalfe C, King M. Evaluating the use of benzodiazepines following recent bereavement. Br J Psychiatry 2001;178: 36-41.[Abstract/Free Full Text]
13. Ben Zion IZ, Meiri G, Greenberg BD, Murphy DL, Benjamin J. Enhancement of CO₂-induced anxiety in healthy volunteers with the serotonin antagonist metergoline. Am J Psychiatry 1999;156: 1635-7.[Abstract/Free Full Text]
14. Himle JA, Abelson JL, Haghightgou H, Hill EM, Nesse RM, Curtis GC. Effect of alcohol on social phobic anxiety. Am J Psychiatry 1999;156: 1237-43.[Abstract/Free Full Text]
15. Stoll AL, Severus WE, Freeman MP, Rueter S, Zboyan HA, Diamond E, et al. Omega 3 fatty acids in bipolar disorder: a preliminary double-blind, placebo-controlled trial. Arch Gen Psychiatry 1999;56: 407-12.[Abstract/Free Full Text]
16. Heresco-Levy U, Javitt DC, Ermilov M, Mordel C, Silipo G, Lichtenstein M. Efficacy of high-dose glycine in the treatment of enduring negative symptoms of schizophrenia. Arch Gen Psychiatry 1999;56: 29-36.[Abstract/Free Full Text]
17. Schneier FR, Goetz D, Campeas R, Fallon B, Marshall R, Liebowitz MR. Placebo-controlled trial of moclobemide in social phobia. Br J Psychiatry 1998;172: 70-7.[Abstract/Free Full Text]
18. Young SA, Hurt PH, Benedek DM, Howard RS. Treatment of premenstrual dysphoric disorder with sertraline during the luteal phase: a randomised, double-blind, placebo-controlled crossover trial. J Clin Psychiatry 1998;59: 76-80.[ISI][Medline]
19. Sackett DL, Gent M. Controversy in counting and attributing events in clinical trials. N Engl J Med 1979;301: 1410-2.[ISI][Medline]
20. Sheiner LB, Rubin DB. Intention-to-treat analysis and the goals of clinical trials. Clin Pharmacol Ther 1995;57: 6-15.[CrossRef][ISI][Medline]
21. Schulz KF, Grimes DA, Altman DG, Hayes RJ. Blinding and exclusions after allocation in randomised controlled trials: survey of published parallel group trials in obstetrics and gynaecology. BMJ 1996;312: 742-4.[Abstract/Free Full Text]

(Accepted 11 November 2003)

Related Articles

Future of psychotherapy in the NHS: Control groups play important part

D B Double
BMJ 2004 329: 514. [Extract] [Full Text]

Turning a blind eye: Testing the success of blinding and the CONSORT statement

Douglas G Altman, Kenneth F Schulz, and David Moher
BMJ 2004 328: 1135. [Extract] [Full Text]

Turning a blind eye: Authors have blinkered view of blinding

Stephen J Senn
BMJ 2004 328: 1135-1136. [Extract] [Full Text]

Turning a blind eye: Why we don't test for blindness at the end of our trials

David L Sackett
BMJ 2004 328: 1136. [Extract] [Full Text]

Evidence of success of blinding in clinical trials is scant
BMJ 2004 328: 0. [Full Text]

This article has been cited by other articles:

• Sackett, D. L (2007). Commentary: Measuring the success of blinding in RCTs: don't, must, can't or needn't?. Int J Epidemiol 36: 664-665 [Full text]  
• Hrobjartsson, A, Forfang, E, Haahr, M., Als-Nielsen, B, Brorson, S (2007). Blinded trials taken to the test: an analysis of randomized clinical trials that report tests for the success of blinding. Int J Epidemiol 36: 654-663 [Abstract] [Full text]  
• Poolman, R. W., Struijs, P. A.A., Krips, R., Sierevelt, I. N., Marti, R. K., Farrokhyar, F., Bhandari, M. (2007). Reporting of Outcomes in Orthopaedic Randomized Trials: Does Blinding of Outcome Assessors Matter?. JBJS 89: 550-558 [Abstract] [Full text]  
• Vickrey, B. G., Mittman, B. S., Connor, K. I., Pearson, M. L., Della Penna, R. D., Ganiats, T. G., DeMonte, R. W. Jr., Chodosh, J., Cui, X., Vassar, S., Duan, N., Lee, M. (2006). The Effect of a Disease Management Intervention on Quality and Outcomes of Dementia Care: A Randomized, Controlled Trial.. ANN INTERN MED 145: 713-726 [Abstract] [Full text]  
• Bausell, R. B., Lao, L., Bergman, S., Lee, W.-L., Berman, B. M. (2005). Is Acupuncture Analgesia an Expectancy Effect?: Preliminary Evidence Based on Participants' Perceived Assignments in Two Placebo-Controlled Trials. Eval Health Prof 28: 9-26 [Abstract]  
• Matthews, S B, Waud, J P, Roberts, A G, Campbell, A K (2005). Systemic lactose intolerance: a new perspective on an old problem. Postgrad. Med. J. 81: 167-173 [Abstract] [Full text]  
• Double, D B (2004). Future of psychotherapy in the NHS: Control groups play important part. BMJ 329: 514-514 [Full text]  
• Altman, D. G, Schulz, K. F, Moher, D. (2004). Turning a blind eye: Testing the success of blinding and the CONSORT statement. BMJ 328: 1135-1135 [Full text]  
• Senn, S. J (2004). Turning a blind eye: Authors have blinkered view of blinding. BMJ 328: 1135-1136 [Full text]  
• Sackett, D. L (2004). Turning a blind eye: Why we don't test for blindness at the end of our trials. BMJ 328: 1136-1136 [Full text]  

Rapid Responses:

Read all Rapid Responses

A Blinkered view of Blinding

Stephen J Senn
bmj.com, 20 Feb 2004 [Full text]

Why we don't test for blindness at the end of our trials.

David L. Sackett
bmj.com, 20 Feb 2004 [Full text]

Changing the mindset about unblinding in clinical trials

D B Double
bmj.com, 21 Feb 2004 [Full text]

Testing the success of blinding and the CONSORT Statement

Douglas G. Altman, et al.
bmj.com, 21 Feb 2004 [Full text]

The Blind leading the Blind?

Michael Goodyear
bmj.com, 24 Feb 2004 [Full text]

Widening the field of vision

Stan Shapiro, et al.
bmj.com, 24 Feb 2004 [Full text]

Turning a blind eye to bias in clinical trials

D B Double
bmj.com, 26 Feb 2004 [Full text]

Blind evaluation of blinding process

Santosh K Chaturvedi
bmj.com, 2 Mar 2004 [Full text]

Failure to assess blinding can make clinical trials hard to interpret

Andrei SP Brennan, et al.
bmj.com, 23 Mar 2004 [Full text]

The Paradox of the Placebo

Raywat S. Deonandan, et al.
bmj.com, 16 Jun 2004 [Full text]

Placebo-controlled: is it really a MeSH term?

Jafar Kolahi
bmj.com, 29 Sep 2006 [Full text]