Turning a blind eye: the success of blinding reported in a random sample of randomised, placebo controlled trials
BMJ 2004; 328 doi: https://doi.org/10.1136/bmj.328.74327.37952.631667.EE (Published 19 February 2004) Cite this as: BMJ 2004;328:432- Dean Fergusson, scientist (dafergusson{at}ohri.ca)1,
- Kathleen Cranley Glass, professor2,
- Duff Waring, research associate4,
- Stan Shapiro, professor3
- 1Ottawa Health Research Institute, Clinical Epidemiology Program, 501 Smyth Road, Box 201, Ottawa, ON, Canada K1H 8L6
- 2Departments of Human Genetics Pediatrics and Biomedical Ethics Unit, McGill University, Montreal, QC, Canada
- 3Department of Epidemiology and Biostatistics, McGill University
- 4Research Ethics and Regulation Group, Faculty of Law, University of Toronto, Toronto, ON, Canada
- Correspondence to: D Fergusson
- Accepted 11 November 2003
Abstract
Objective To examine the reporting and success of double blinding in a sample of randomised, placebo controlled trials from leading general medicine and psychiatry journals.
Methods Identification of placebo controlled, randomised controlled trials from prespecified general medical and psychiatric journals indexed on Medline between 1 January 1998 and 1 October 2001, from which a random sample of 200 randomised clinical trials was chosen, of which 191 trials were evaluated.
Results Only seven of the 97 (7%) general medicine trials provided evidence on the success of blinding, with five reporting that the success of blinding was imperfect. In trials from psychiatric journals, the success of blinding was reported in eight of the 94 trials, with four reporting that the blinding was imperfect. Overall, only four of the 191 (2%) trials assessed blinding in the participants and either the outcome assessors or the investigators.
Conclusions The current lack of reporting on the success of blinding provides little evidence that success of blinding is maintained in placebo controlled trials. Trialists and editors should make a concerted effort to incorporate, report, and publish such information and its potential effect on study results.
Introduction
Although the definition of double blind varies,1 we consider a trial to be double blind when the patient, investigators, and outcome assessors are unaware of the patient's assigned treatment throughout the conduct of the trial.2 Placebos are commonly used as an inactive treatment to achieve double blinding. Active placebos, with which symptoms or side effects are imitated, can also be used. Placebos are justly used when no existing effective treatment is available. Sometimes, placebos are proposed instead of a standard existing treatment or standard care to ensure assay sensitivity. That is, to demonstrate the effectiveness of a new treatment, it must be demonstrated against a “clean” control. The argument is that although the new treatment may be found to be as effective or more effective than standard treatment in a clinical trial, both treatments may very well be ineffective. Assaysensitivity is the ability of a trial to distinguish effective interventions from ineffective interventions. It depends on the effect size that is to be detected. As such, the investigators need to know the anticipated effects of the control intervention. It is argued that placebos are the ideal choice as their anticipated benefits are known to be marginal. This argument is predicated on the belief that participants, investigators, and outcome assessors remain blinded to the treatment assignment. If the blinding of the placebo arm is not effective then the protection against expectation effects, biased assessment, contamination, and co-intervention are all lost. The observed superiority of a new treatment over placebo could merely be a consequence of loss of this control—and an ineffective new treatment would spuriously seem to be superior. Because of the importance of the success of blinding, the Consolidated Standards for Reporting of Trials (CONSORT) Group has explicitly incorporated the issue. Section 11(b) of the CONSORT statement states that the success of blinding is to be reported in the publication.3
It is not sufficient that trials describe themselves as double blind. It is also important thatthe efficacy of the blinding is actually assessed. In other words, an assessment of the face validity of the double blinding is needed. To assess the reporting and success of double blinding, we chose a random sample of randomised, placebo controlled trials from leading journals in general medicine and psychiatry. Although we have focused on placebo controlled trials, the issues discussed also arise in double blind trials with active controls.
Methods
For this study we selected five of the top general medical journals (JAMA, New EnglandJournal of Medicine, BMJ, Lancet, and Annals of Internal Medicine) and four of the top journals in psychiatry (Archives of General Psychiatry, Journal of Clinical Psychiatry, British Journal of Psychiatry, and American Journal of Psychiatry). Our Medline search used publication type “randomised controlled trial” and the MeSH term “placebo-controlled” to identify placebo controlled randomised trials that were indexed on Medline between the dates of 1 January 1998 and 1 October 2001 and published in these nine journals. All citations from this search were then consecutively numbered, and a random number generator was used to select 100 trials from the general medicine journals and 100 trials from the psychiatry journals. We reasoned that 100 trials from each discipline was a manageable number to abstract and anadequate number to obtain a good estimate of the number of trials reporting the success of blinding, and we performed no formal sample size calculations.
Data abstraction forms were developed and included document identification, the type of interventions, type of placebo, matching characteristics of placebo to intervention, who was blinded, andthe evidence of successful blinding. A trial indicating that a “similar” placebo was used but did not specify how it was similar was scored as “not mentioned” (our rationale is that the term “similar” is vague and therefore inadequate). The page number and location for each data item was also recorded. Six people abstracted all the data. At least two people independently abstracted data for each study. Either consensus or a third party resolved any differences. All data were entered into an electronic database (Microsoft Excel 2000).
Results
The Medline search identified a total of 473 randomised controlled trials in the general medicine literature and 192 trials in the psychiatric literature, from which we randomly chose 100 trials from general medicine and 100 trials from psychiatry. Nine trials were removed from further analysis as they were not placebo controlled trials despite being identified as such in the systematic literature search. Thus, we evaluated 97 trials from the general medicine literature and 94 trials from psychiatry literature.
General medicine
Table 1 provides information on the type of interventions and placebos used in the 97 trials in general medicine. Eighty three per cent of all interventions were pharmacological. Nutritional supplements (9%) were the second most frequent intervention used. Sixteen of the 97 trials did not report the type of placebo used. Of trials that reported the type of placebo, an injection (either subcutaneous or intramuscular) was the most common (23; 27%), followed by tablet (20; 24%) and capsule (18; 21%).
The matching characteristics of placebo to the intervention was reported in 51 (53%) of the trials; one trial (1%) also reported the dissimilarity between placebo and intervention (table 2). Appearance was the characteristic most often reported by investigators (46 of 51 trials), followed by taste (9 of 51 trials).
Only seven of the 97 trials (7%) provided evidence on the success of blinding (table 3).4–10 All seven trials assessed the success of blinding in study participants. One trial assessed the success of blinding in individuals assessing study outcome.6 All seven trials presented a method for assessing blinding. Five of the trials presented blinding data for each trial arm, one trial presented overall aggregated data only, and one trial provided no data. Five trials reported that the success of blinding was imperfect.4 6–8 10 The trial that did not present blinding data described blinding as successful without further comment.9 The trial that reported aggregated blinding data did not comment, qualitatively, or provide statistical tests of success of blinding.5
Psychiatry
Most psychiatry trials used pharmacological interventions (table 1). Over 40% of the trials did not report the type of placebo used. Of the placebos reported, 78% were either a capsule or tablet.
The matching characteristics between intervention and placebo were reported in 30 (32%) of trials, with appearance being the most often reported (table 2). Eight of the 94 trials reported evidence on successful blinding (table 4).11–18 Of these, six assessed the success of blinding in patients.12 14–18 Two studies provided blinding data for both subjects and outcome assessors16 17; one study reported blinding data for both treatment administrators and outcome assessors,11 and one study provided data for treatment administrators only.13 Six of the eight studies presented a method for blinding assessment in the Methods or Results section of the article and the other two presented it in the Discussion section. Four of the trials presented blinding data broken down by treatment allocation12 14 15 17; one trial presented aggregated data and did not provide data broken down by treatment allocation13; and two presented no data on blinding.11 16 Of the eight trials, the blinding was reported as less than optimal in four.11 14 15 18
Discussion
The quality of reporting in clinical trials has evolved. Over the years, trialists have been held more accountable and responsible for the quality of trial reporting. This evolution began with the need for reporting the numbers of patients screened, enrolled, randomised, and analysed,19 and progressed to the reporting of patient withdrawals and its importance for the analysis and interpretation of study results.20 Building on this progress, there is a need for trialists and journals routinely to report the methods of blinding and the subsequent success of this blinding.21
Our examination of the success of blinding challenges the notion that placebo controlled trialsinherently possess assay sensitivity. Clearly, there is a failure among investigators and journalsin reporting the success of blinding. Only 15 of the 191 trials (8%) provided such information, be it qualitative or quantitative. Of the 15 trials, only five trials reported that blinding was successful,9 1213 16 17 and of these, three did not present any quantitative data analysis to supporttheir claim.9 13 16
Only four trials assessed blinding in both the participants and either the outcome assessors orthe investigators.6 12 16 17 Thus, the face validity of the double blinding was only reported in four of the 191 articles (2%). This deficiency in reporting translates into a paucity of evidence that a placebo ensures a “clean” control. Furthermore, the quality of evidence in the few studies that reported on the success of blinding is weak on two fronts: the quality of the data and the evidence that blinding was successful. The success of blinding was described as less than optimal in nine of the 14 trials that reported on blinding, and of the five trials that reported that blinding was maintained, only two provided data to support their claim.12 17 Unfortunately, when we examined the data and analysis provided by these two trials we found that their claim of success is debatable.
We would like to see Item 11b of CONSORT revised to require the assessment of blinding for all double blind randomised trials. Trialists have an ethical responsibility to justify the use of a placebo for blinding purposes in their research protocol and informed consent procedures. Thus, it seems reasonable to suggest that an assessment of the success of blinding is necessary. If blinding is not assessed, we may delude ourselves as to exactly what information we gain from incorporating a placebo comparison. Although all trials should assess blinding, the types of trials that willparticularly benefit are trials with subjective outcomes or outcomes reported by patients (for example, quality of life instruments), or trials where the side effects are well known. Even though there may be problems with analysing and interpreting the results of success, this does not providea rationale for not doing it. Clearly, the lack of successful blinding can bias observed estimatesof effect. Although this bias is differential, its direction may not be easily ascertained. We might anticipate that evidence of unsuccessful blinding in a “double blind” active versus placebo trial would result in a positive bias and hence lead to an overestimate of the treatment effect. However, unblinded patients receiving placebo may seek other treatments, especially if there is established effective treatment available, and this makes the extent and even the direction of bias difficult to determine.
We believe that trialists need to report a minimum set of information. This includes the countsof all patients allocated to each treatment; the counts of patients who guess treatment assignmentby the group to which they were allocated; the counts of correct guesses and those who are undecided; the analytical methods and results used to assess success of blinding; and the author's interpretation of the efficacy of blinding and the effect on study results. The data abstracted for thisstudy show a substantial lack of reporting with respect to these minimum, essential items, as illustrated by the number of vacant fields in tables 3 and 4.
What is already known on this topic
Placebo controls are commonly used in randomised trials to blind investigators, outcome assessors, and patients to treatment assignment
Placebo controls have been advocated instead of existing effective treatment because they ensure assay sensitivity
Unsuccessful double blinding results in a differential bias of effect measures
What this study adds
The success of blinding is not well reported
The success of blinding in trials that do report is often poor
Little evidence exists that placebos provide assay sensitivity
The current lack of reporting on the success of blinding provides little evidence that success of blinding is maintained in placebo controlled trials. Trialists and editors need to make a concerted effort to incorporate, report, and publish such information and its potential effect on studyresults. The efficacy of the blinding cannot be assumed on theoretical grounds. We need evidence before we can assert that assay sensitivity exists in randomised, double blind, placebo controlled trials.
Acknowledgments
We thank Julie Comber and Jennifer Marshall for article retrieval and data collection.
Contributors: DF, KG, DW, and SS conceived and designed the study. DF collected, managed, and analysed the data. All authors interpreted the data and wrote the paper. DF is the guarantor.
Footnotes
-
Funding This work was funded in part by the Canadian Institutes of Health Research
-
Competing interests None declared
-
Ethical approval: Not required.