- Kenneth F Schulz, assistant directora,
- David A Grimes, professorb,
- Douglas G Altman, headc,
- Richard J Hayes, headd
- a Division of Sexually Transmitted Diseases Prevention, Centers for Disease Control and Prevention, Atlanta, GA 30333, USA
- b Department of Obstetrics, Gynecology, and Reproductive Sciences, University of California, San Francisco, CA 94110, USA
- c Medical Statistics Laboratory, Imperial Cancer Research Fund Medical Statistics Group, Centre for Statistics in Medicine, Oxford OX3 7LF
- d Tropical Health Epidemiology Unit, London School of Hygiene and Tropical Medicine, London WC1E 9HT
- Correspondence to: Dr Schulz.
- Accepted 4 January 1996
Objective: To assess the methodological quality of approaches to blinding and to handling of exclusions as reported in randomised trials from one medical specialty.
Design: Survey of published, parallel group randomised controlled trials.
Data sources: A random sample of 110 reports in which allocation was described as randomised from the 1990 and 1991 volumes of four journals of obstetrics and gynaecology.
Main outcome measures: The adequacy of the descriptions of double blinding and exclusions after randomisation.
Results: Though 31 trials reported being double blind, about twice as many could have been. Of the 31 trials only eight (26%) provided information on the protection of the allocation schedule and only five (16%) provided some written assurance of successful implemention of double blinding. Of 38 trials in which the authors provided sufficient information for readers to infer that no exclusions after randomisation had occurred, six (16%) reported adequate allocation concealment and none stated that an intention to treat analysis had been performed. That compared with 14 (27%) and six (12%), respectively, for the 52 trials that reported exclusions.
Conclusions: Investigators could have double blinded more often. When they did double blind, they reported poorly and rarely evaluated it. Paradoxically, trials that reported exclusions seemed generally of a higher methodological standard than those that had no apparent exclusions. Exclusions from analysis may have been made in some of the trials in which no exclusions were reported. Editors and readers of reports of randomised trials should understand that flawed reporting of exclusions may often provide a misleading impression of the quality of the trial.
Poor reporting of methods used for double blinding raises concerns about how effective blinding was in many of the studies
Some investigators may have excluded some patients from the analysis but not reported these exclusions in their published paper
Given the apparent failure to report exclusions, some of the more biased trials may be mistaken as unbiased and vice versa
Investigators must pay more attention to executing and reporting their approaches to blinding and the handling of exclusions
Randomisation, avoidance of exclusions after trial entry, and double blinding (or masking) may represent the most important methodological components for reducing bias in controlled trials.1 2 3 Indeed, a recent analysis of 250 trials from 33 meta-analyses showed that lack of allocation concealment and double blinding were associated with exaggerated estimates of effects of treatment.4 Those associations provide empirical evidence of bias. The analysis did not, however, find an association between reporting exclusions after randomisation and exaggerated estimates.4 That implies that some authors of the trials may not have accurately reported those exclusions that had actually taken place.4
Though investigations have detailed the poor quality of reporting of randomisation in recently published randomised controlled trials,5 6 7 8 few studies have considered other methodological components. We therefore assessed the level of reporting on double blinding and handling of exclusions by evaluating 110 reports of randomised controlled trials published during 1990-1 in four journals of obstetrics and gynaecology. We also examined the possibility that some reports of trials ostensibly without exclusions may have actually had exclusions.
We evaluated reports from four journals of obstetrics and gynaecology: the American Journal of Obstetrics and Gynecology, the British Journal of Obstetrics and Gynaecology, Obstetrics and Gynecology, and the Journal of Obstetrics and Gynaecology. We identified all 206 reports of randomised trials published in the 1990 and 1991 volumes of those journals.5 We included articles in which authors purported to allocate individual subjects randomly to parallel (uncrossed) groups.5 From those 206 reports, we identified 110 for evaluation by selecting all 20 from the Journal of Obstetrics and Gynaecology and by taking a random sample (computer random number generator9) of 30 from each of the three other journals.
We developed a data collection instrument to assess blinding and exclusions. For consistency of measurement one investigator (KFS) did all of the initial assessments. To examine the reproducibility of items on our questionnaire another investigator (DAG) unaware of the initial assessments independently assessed a random sample of 10 trials. For this study we analysed only items that exhibited consistency between assessors. We linked the information on allocation concealment from the earlier research5 to the records from this study.
An “allocation concealment” process seeks to prevent selection bias, protects the assignment sequence before and until allocation, and can always be implemented.5 In contrast with allocation concealment, double blinding seeks to prevent ascertainment bias, protects the sequence after allocation, and cannot always be implemented.5 A “double blind” trial shields participants, caregivers, and outcome evaluators from knowledge of assignment to particular treatments.10 11 Exclusions after randomisation can occur because of eventual discovery of ineligibility of a participant, deviations from protocol, withdrawals, or losses to follow up. The valid, unbiased approach analyses all randomised participants in the originally assigned groups, regardless of compliance with protocol (“intention to treat analysis”).10 11 12 13 14 15 16 17 18
Most of the trials in this study (65%; 72/110) entailed pharmaceutical interventions. Overall, authors reported 31 (28%) trials as double blind, 15 (14%) as having blinded outcome assessments, one (1%) as having blinded the participants or caregivers, and 63 (57%) as not having any form of blinding. We judged double blinding to be feasible in 65 trials. Thus, the authors reported double blinding in only 48% of trials that could have been double blinded. In these 31 double blind trials, authors specified the intervention medium as capsules in three (10%), tablets in 11 (35%), injection or intravenous administration in 11 (35%), double dummy11 in two (6%), and various other approaches—for example, sprays, creams—in four (13%). Only 14 (45%) reports described similarity of the treatment and control regimens—for example, appearance, taste, administration.
Only eight of the 31 reports (26%) provided information on the protection of the allocation schedule. Those statements included keeping the code in a secure location or not breaking the code until the end of the study. Only five (16%) provided statements that blinding had been implemented successfully. Investigators reported testing the efficacy of blinding in only two of the 31 trials. Both found substantial unblinding of the assignments.
Authors provided insufficient information on the number of participants randomised and analysed even to infer whether exclusions had taken place in nine (8%) reports. In 52 (47%) the authors indicated exclusion of at least one participant: of these, 29 reports indicated exclusions of more than 10% of randomised participants. Although all 52 reports provided reasons for exclusions, only 34 provided reasons by assignment to treatment group. Four reports stipulated that the exclusions occurred before the assignment code was broken.
In 49 trials (45%) authors reported no apparent exclusions (table 1). Only 11 of these, however, explicitly stated that no exclusions had occurred after randomisation. Authors in the 38 other trials simply reported analysing the same number of participants as randomised, thereby implying that no exclusions had occurred. In these 38 reports with no apparent exclusions authors less commonly reported an intention to treat analysis (P=0.037; two tailed Fisher's exact test9) and less commonly reported adequate allocation concealment (relative risk 0.59; Taylor series 95% confidence interval9 0.25 to 1.39) than did authors in the 52 trials with exclusions. Though the 95% confidence interval for concealment overlaps 1.0, it does not overlap the reverse results we expected from trials without exclusions. Indeed, the 38 trials with no apparent exclusions less commonly indicated adequate allocation concealment than the 11 trials that explicitly stated no exclusions (relative risk 0.35; 0.13 to 0.92).
Only about half of the trials that could have been double blinded were done double blind. Because double blinding reduces bias4 these investigators should have used it more often. Double blinding, however, may not always be appropriate. In certain trials, particularly those with objective end points such as death, the anticipated gain in reduction of bias may not always be worth the additional difficulty and cost. Furthermore, in other trials—for example, surgical trials—double blinding becomes difficult or impossible. Nevertheless, even when double blinding is not feasible, blinding of assessment is advisable and often possible.
For double blinding to be successful, investigators must use and report effective methodological procedures.3 Because of scanty reporting we question the effectiveness of many of these attempts at double blinding. Indeed, when investigators tested blinding in two trials they found failures. Yet a third trial reported assigning participants by hospital number and giving two drugs to one group and only one drug to the other. Such deficient approaches thwart successful blinding. Thus, blinding seems compromised in at least three trials, although we suspect that such problems existed more widely.
Our results are consistent with other findings. Authors have often just reported their study as “double blind” and not provided much further clarifying information.19 20 21 Though investigators have reported compromised blinding,22 23 such candid reporting seems rare. We believe that authors should provide the following information on double blind trials: (a) mechanism (for example, capsules, tablets); (b) similarity of characteristics of treatment (for example, appearance, taste, administration); (c) control of allocation schedule (for example, the location of the schedule during the trial, when the code was broken for the analysis, and circumstances under which the code could be broken for individual cases); and (d) some statement on the perceived success (or failure) of the double blinding efforts. Ideally this last guideline would include tests of efficacy, but such tests may have problems with validity. Clearly, information on allocation concealment5 should also be provided. Hopefully these guidelines will provoke further dialogue on requirements for implementing and reporting double blinding.3
An intention to treat analysis uniquely preserves the qualities of bias reduction of randomisation. Improper handling of exclusions after randomisation introduces bias.10 11 12 13 14 15 16 17 18 19 Unfortunately, many trials in our study provided inadequate information on exclusions, which corroborates previous research.20 21 24 Authors must provide complete information on exclusions after randomisation and whether the primary analysis used the intention to treat principle.3 If authors present other analyses—for example, per protocol17—details should be provided in the report. While not unbiased, those other analyses may prove useful and illuminating.17
Investigators sufficiently diligent to produce a trial without exclusions would also seemingly be more likely to produce a trial with proper allocation and analysis. That may have happened in those trials in which authors explicitly reported no exclusions but not in those with no apparent exclusions. We suggest that some investigators who do not report exclusions may in fact have made exclusions during the trial but ignored them in the report. That supports earlier findings.4 Indeed, two trials with documented exclusions have published reports that indicate no exclusions.25
Bias due to exclusions may exist in some of those trials that report no apparent exclusions, while many comparatively better trials may come from those in which authors reported exclusions. Some readers, however, may view trials without exclusions as less biased. Thus, some of the more biased trials may be mistaken as unbiased and many of the less biased trials as biased. Until authors comprehensively report exclusions after randomisation, editors, readers, and those conducting systematic reviews should all be wary of this disquieting paradox.
Funding Division of Sexually Transmitted Diseases Prevention, Centers for Disease Control.
Conflict of interest None.