BMJ  2005;330:1049 (7 May), doi:10.1136/bmj.38414.422650.8F (published 7 April 2005)

Paper

Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study

¹ Nordic Cochrane Centre, H:S Rigshospitalet, DK-2100 Copenhagen, Denmark, ² Department of Medicine, University of Toronto, ³ Cancer Research UK Medical Statistics Group, Centre for Statistics in Medicine, Oxford OX3 7LF

Abstract

Objectives To compare how allocation concealment is described in publications of randomised clinical trials and corresponding protocols, and to estimate how often trial publications with unclear allocation concealment have adequate concealment according to the protocol.

Design Cohort study of 102 sets of trial protocols and corresponding publications.

Setting Protocols of randomised trials approved by the scientific and ethical committees for Copenhagen and Frederiksberg, 1994 and 1995.

Main outcome measures Frequency of adequate, unclear, and inadequate allocation concealment and sequence generation in trial publications compared with protocols, and the proportion of protocols where methods were reported to be adequate but descriptions were unclear in the trial publications.

Results 96 of the 102 trials had unclear allocation concealment according to the trial publication. According to the protocols, 15 of these 96 trials had adequate allocation concealment (16%, 95% confidence interval 9% to 24%), 80 had unclear concealment (83%, 74% to 90%), and one had inadequate concealment. When retrospectively defined loose criteria for concealment were applied, 83 of the 102 trial publications had unclear concealment. According to their protocol, 33 of these 83 trials had adequate allocation concealment (40%, 29% to 51%), 49 had unclear concealment (59%, 48% to 70%), and one had inadequate concealment.

Conclusions Most randomised clinical trials have unclear allocation concealment on the basis of the trial publication alone. Most of these trials also have unclear allocation concealment according to their protocol.

Introduction

Around 44% to 93% of publications of randomised controlled trials lack a clear description of allocation concealment.¹ Empirical studies have shown that publications of trials in which allocation concealment is unclear or inadequate exaggerate the treatment effect by 20-30%, on average.^2-8 Generation of a truly random sequence is an interrelated issue for which there is suggestive empirical evidence of an associated inflation of the treatment effect if the trial publication does not document adequate procedures.^3-5 We compared how allocation concealment is described in publications of randomised clinical trials and corresponding protocols, and we estimated how often trial publications with unclear allocation concealment have adequate concealment according to the protocol.

Methods

Our cohort consisted of all published randomised trials with protocols approved by the scientific and ethical committees for Copenhagen and Frederiksberg in 1994 and 1995. We identified trial publications by contacting the principal investigators and by searching PubMed, Embase, and the Cochrane central register of controlled trials.⁹

Outcomes and data extraction
Our outcome measures were frequency of adequate, unclear, and inadequate allocation concealment and sequence generation in trial publications compared with protocols; the proportion of protocols where methods were reported to be adequate when the trial publications gave unclear descriptions; the type and frequency of methods used for allocation concealment; and the prevalence of other trial characteristics that might undermine concealment.

Assessment of adequacy of allocation concealment
We considered the following methods for allocation concealment as adequate:^{2-5 10-12} central randomisation; numbered coded vehicles; opaque, sealed, and sequentially numbered envelopes; and other convincing methods (see table A on bmj.com). Inadequate methods concerned open or predictable sequences of allocation—for example, date of birth, case record number or similar, and open tables of random numbers. Studies were categorised as unclear that did not fall into one of these categories or that did not provide information.

To ensure consistency and transparency and to capture how strict application compared with loose application of our criteria might influence our results, we operationalised our interpretations of authors' descriptions of allocation concealment (see table A on bmj.com). The strict criteria are those recommended for Cochrane reviews,¹⁰ except for an elaboration on centralised randomisation. The loose criteria, defined retrospectively, comprised the most liberal criteria used in any of the empirical studies of bias associated with unclear or inadequate allocation concealment.^2-8

Assessment of adequacy of sequence generation
Adequate methods of sequence generation included computer generated random numbers, tables of random numbers, or drawing lots or envelopes. Inadequate methods could be related to prognosis such as date of birth or year of admission. Unclear methods were methods not falling into one of these two categories or where the methods were not described.

Results

Allocation concealment
Using the strict criteria, 96 of the 102 trials (94%, 95% confidence interval 88% to 98%) had an unclear allocation concealment according to their publications; according to the protocols, 15 of these 96 trials (16%, 9% to 24%) had adequate allocation concealment and one had inadequate concealment, whereas most (80 of 96; 83%, 74% to 90%) had unclear concealment (table).

View this table: [in this window] [in a new window]   Adequacy of allocation concealment as described in pairs of protocols and corresponding trial publications in 102 trials according to strict and loose criteria. Values are numbers

 

Using the loose criteria, 83 of the 102 trials had unclear allocation concealment (81%, 72% to 88%). According to the protocols, 33 of these 83 publications (40%, 29% to 51%) had adequate allocation concealment, one had inadequate concealment, and 49 (59%; 48% to 70%) had unclear concealment (see bmj.com).

According to the strict criteria, 20 of the 102 studies described adequate allocation concealment (five publications and 19 protocols; see bmj.com). When the loose criteria were applied, 51 studies (18 publications and 45 protocols) described adequate concealment.

Sequence generation
Eighty one of the 102 trial publications gave no information on how the allocation sequence was generated; 16 of these 81 trials (20%; 12% to 30%) described adequate sequence generation in the protocol. No protocols or trial publications reported inadequate methods of sequence generation.

Methods used for allocation concealment
See bmj.com for methods used for allocation concealment. Numbered coded vehicles was the most frequently applied method according to the protocols (26 of 102) but had the lowest rate of appearance in the trial publications (three of 26). None of the 17 trials using central randomisation fulfilled the strict criteria, as none described concealment of the randomisation sequence from the central staff, only four described irreversibility of the treatment assignment, and none described that prognostic data irrelevant to stratification must not be revealed to the central office (in three trials such data were positively requested). In 39 of the 102 trials neither the protocols nor the publications provided any information on attempts to conceal the allocation. In four trials, the protocol and the publication gave conflicting information on which method was used.

Trial characteristics that might weaken an otherwise adequate allocation concealment regimen
Block randomisation
In 14 trials, block randomisation could partly have compromised allocation concealment because the block size was explicitly stated in the protocol. This is problematic since a known block size enables qualified guesswork to predict upcoming allocations towards the end of the block.

Tasks that should not be carried out by the same party
The preparation of envelopes for concealment was described in the passive tense in nine of the 13 studies (see bmj.com). Thus it is unknown whether the same person prepared the envelopes, enrolled the patients, and administered the envelopes, particularly as seven of the nine studies were single centre studies.

Code envelopes
In 42 of the 55 double blind studies, a security system for emergency code breaking was described in the protocol but mentioned in only one publication. Overall, 90% (38 of 42) of these protocols specified that envelopes or a similar system would be present at the clinical location. Deciphering the contents of such envelopes, for instance by holding them against strong light, might have revealed the allocation for the next patient; yet only one of the 38 protocols (3%) described the code envelopes as opaque.

Discussion

Most trial publications provided unclear information on allocation concealment. When we applied strict criteria the corresponding protocols clarified that 16% had adequate concealment compared with 40% when we applied loose criteria. Thus, regardless of the criteria applied, most of the protocols also provided unclear information or gave rise to additional concern that the allocation concealment might have been compromised. A similar pattern of insufficient reporting was found for sequence generation.

Our results make it reasonable to assume that the empirical surveys, which show a 20-30% exaggeration of the treatment effect for trial publications with unclear or inadequate allocation concealment, included some trials with allocation concealment that was adequately carried out but insufficiently reported.^2-5 This implies that if inadequate concealment with ensuing selection bias is to explain the observed exaggeration in the previous studies,^2-5 then an even larger exaggeration would be expected for those trials where neither the publication nor the protocol indicated adequate concealment.

The strength of our study is that it is the first account of how allocation concealment is described in a representative cohort of trial protocols and subsequent publications. The detailed data extraction allowed for sensitivity analysis of the strictness of the applied criteria and for finding additional elements that could compromise concealment.

One limitation is that even in the cases where the protocols provided explicit descriptions of allocation concealment, the assumption that the trials were conducted according to the protocol, might not always be true.⁹ Another limitation is that it is still unresolved as to what extent the exaggeration associated with unclear allocation concealment in trial publications can be explained by inadequate concealment and ensuing selection bias, as opposed to unclear concealment being a marker of other sources of bias.³

Our strict criteria might have been too stringent, and four related studies used criteria with a stringency somewhere between our strict and loose criteria.^13-16

Three studies indicated that trial publications with unclear allocation concealment reflect poor reporting of adequate methods, rather than poor methods.^13-15

In a retrospective questionnaire survey of investigators by Hill et al, 78% of 32 trials with unclear allocation concealment in trial publications were adequately concealed according to the primary investigators.¹³ The finding, however, centred on a small sample, on the reliability and memory of the investigators, and on assumptions of what the 20% of non-responders would have replied.

Devereaux et al found that 54 of 56 trials with unclear allocation concealment in the trial publication were adequately concealed according to a preannounced telephone interview of the investigators.¹⁴ These trials were published in journals with higher impact factors than ours and might be of higher methodological quality. Or maybe some of the protocols in our cohort failed to adequately detail all the procedures adopted to protect against bias. The reliability of surveyed investigators has previously been reported on in two surveys where 86% (42 of 49) and 80% (28 of 35) of investigators denied the existence of unreported outcomes, although there was evidence to the contrary in their study protocols.^{9 17}

Another survey was done on trials carried out within the framework of the Radiation Therapy Oncology Group, where all trial protocols undergo a rigorous six step peer review process.¹⁵ Although all studies had adequate allocation concealment (central randomisation) only 42% reported such concealment in the trial publication. However, as the authors pointed out, their result has limited generalisability since few trial protocols undergo such rigorous peer review and, as documented in our broad cohort, central randomisation is not the most commonly used method across medical specialties.

Finally, Liberati et al¹⁶ reported results similar to ours; among 47 trials with unclear allocation concealment in the publications, 11 (23%) used adequate randomisation methods (defined as central randomisation) according to a subsequent telephone interview of all but one investigator. The discrepancywith the findings of Hill et al and Devereaux et al might reflect the difference in response rate, criteria for adequate concealment, recentness of the included trials, or the strategies for contacting and phrasing the questions to the investigators.

What is already known on this topic In most trial publications, allocation concealment is unclear or inadequate Unclear or inadequate concealment in publications is associated with an exaggeration of the treatment effect by 20-30%, on average What this study adds Most often allocation concealment is unclear in trial protocols Gatekeepers who sanction protocols should require that adequate methods of allocation concealment be described and used Protocols should be publicly accessible to enhance critical appraisal of trials

It is prudent to assume that a notable fraction of the overestimation of the treatment effect associated with unclear allocation concealment is caused by selection bias. This fraction can be reduced through several mechanisms. Journals should endorse and enforce the consolidated standards of reporting trials statement (www.consort-statement.org), which recommends explicit description of the allocation procedures in publications of trials, and the gatekeepers who sanction protocols for funding and approval should demand that adequate methods are described in protocols and implemented in trials. Furthermore, our study adds to the argument that protocols should be made publicly available,^{9 18 19} because public access would increase the reliability of critical appraisal of the fraction of trials where the protocol does describe methods for allocation concealment.

---

Further details of methods, and examples, are on bmj.com

This is the abridged version of an article that was posted on bmj.com on 7 April 2005: http://bmj.com/cgi/doi/10.1136/bmj.38414.422650.8F

We thank the scientific and ethical committees for Copenhagen and Frederiksberg, who provided access to the trial protocols and offered administrative support; the researchers who responded to the questionnaires; and J Hedegaard for secretarial support.

Contributors: See bmj.com

Funding: JP, AH, EF, and PCG are funded by Copenhagen Hospital Corporation. AWC was supported by the Rhodes Trust. DGA is supported by Cancer Research UK. The funding organisations had no role in any aspect of the study, the manuscript, or the decision to publish.

Competing interests: None declared.

Ethical approval: Not required.

References

1. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001; 134: 663-94.[Abstract/Free Full Text]
2. Jüni P, Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 2001; 323: 42-6.[Free Full Text]
3. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 408-12.[Abstract/Free Full Text]
4. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998; 352: 609-13.[CrossRef][Web of Science][Medline]
5. Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001; 135: 982-9.[Abstract/Free Full Text]
6. Balk EM, Bonis PA, Moskowitz H, Schmid CH, Ioannidis JP, Wang C, et al. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA 2002; 287: 2973-82.[Abstract/Free Full Text]
7. Jüni P, Egger M. Allocation concealment in clinical trials. JAMA 2002; 288: 2407-8.
8. Schulz KF, Altman DG, Moher D. Allocation concealment in clinical trials. JAMA 2002; 288: 2406-7.[Free Full Text]
9. Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004; 291: 2457-65.[Abstract/Free Full Text]
10. Alderson P, Green S, Higgins JPT, ed. Assessment of study quality. Cochrane reviewers' handbook 4.2.2 [Updated Mar 2004];sect 6. In: Cochrane Library, Issue 2. Chichester: Wiley, 2004.
11. Meinert C. Clinical trials. Design, conduct and analysis. Oxford: Oxford University Press, 1986.
12. Berger VW, Bears JD. When can a clinical trial be called `randomized'? Vaccine 2003; 21: 468-72.[CrossRef][Web of Science][Medline]
13. Hill CL, LaValley MP, Felson DT. Discrepancy between published report and actual conduct of randomized clinical trials. J Clin Epidemiol 2002; 55: 783-6.[CrossRef][Web of Science][Medline]
14. Devereaux PJ, Choi PT, El-Dika S, Bhandari M, Montori VM, Schunemann HJ, et al. An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol 2004; 57: 1232-6.[CrossRef][Web of Science][Medline]
15. Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, et al. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ 2004; 328: 22-4.[Abstract/Free Full Text]
16. Liberati A, Himel HN, Chalmers TC. A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol 1986; 4: 942-51.[Abstract/Free Full Text]
17. Chan AW, Krleza-Jeric K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 2004; 171: 735-40.[Abstract/Free Full Text]
18. Jüni P, Rutjes AW, Dieppe PA. Are selective COX 2 inhibitors superior to traditional non steroidal anti-inflammatory drugs? BMJ 2002; 324: 1287-8.[Free Full Text]
19. Chalmers I, Altman DG. How can medical journals help prevent poor medical research? Some opportunities presented by electronic publishing. Lancet 1999; 353: 490-3.[CrossRef][Web of Science][Medline]

(Accepted 2 March 2005)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Articles

Industry attack on academics

Jonathan Gornall
BMJ 2009 338: b736. [Extract] [Full Text]

Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study

Lesley Wood, Matthias Egger, Lise Lotte Gluud, Kenneth F Schulz, Peter Jüni, Douglas G Altman, Christian Gluud, Richard M Martin, Anthony J G Wood, and Jonathan A C Sterne
BMJ 2008 336: 601-605. [Abstract] [Full Text] [PDF]

Delve deeper to find the links

John H Noble, Jr
BMJ 2008 336: 59. [Extract] [Full Text] [PDF]

Allocation concealment is neglected in papers and protocols
BMJ 2005 330: 0. [Full Text]

Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group

Heloisa P Soares, Stephanie Daniels, Ambuj Kumar, Mike Clarke, Charles Scott, Suzanne Swann, and Benjamin Djulbegovic
BMJ 2004 328: 22-24. [Abstract] [Full Text] [PDF]

Are selective COX 2 inhibitors superior to traditional non steroidal anti-inflammatory drugs?

Peter Jüni, Anne WS Rutjes, and Paul A Dieppe
BMJ 2002 324: 1287-1288. [Extract] [Full Text] [PDF]

Systematic reviews in health care: Assessing the quality of controlled clinical trials

Peter Jüni, Douglas G Altman, and Matthias Egger
BMJ 2001 323: 42-46. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

• Bausch, B., Spaar, A., Kleijnen, J., Puhan, M. A. (2009). Quality of randomised trials in COPD. Eur Respir J 34: 1060-1065 [Abstract] [Full text]  
• Clark, N. M., Janz, N. K., Dodge, J. A., Xihong Lin, , Trabert, B. L., Kaciroti, N., Mosca, L., Wheeler, J. R., Keteyian, S. (2009). Heart Disease Management by Women: Does Intervention Format Matter?. Health Educ Behav 36: 394-409 [Abstract]  
• Gornall, J. (2009). Industry attack on academics. BMJ 338: b736-b736 [Full text]  
• Rios, L. P., Odueyungbo, A., Moitri, M. O., Rahman, M. O., Thabane, L. (2008). Quality of Reporting of Randomized Controlled Trials in General Endocrinology Literature. J. Clin. Endocrinol. Metab. 93: 3810-3816 [Abstract] [Full text]  
• Wykes, T., Steel, C., Everitt, B., Tarrier, N. (2008). Cognitive Behavior Therapy for Schizophrenia: Effect Sizes, Clinical Models, and Methodological Rigor. Schizophr Bull 34: 523-537 [Abstract] [Full text]  
• Wood, L., Egger, M., Gluud, L. L., Schulz, K. F, Juni, P., Altman, D. G, Gluud, C., Martin, R. M, Wood, A. J G, Sterne, J. A C (2008). Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 336: 601-605 [Abstract] [Full text]  
• Noble, J. H Jr (2008). Delve deeper to find the links. BMJ 336: 59-59 [Full text]  
• Reddy, M., Gill, S. S., Kalkar, S. R., Wu, W., Anderson, P. J., Rochon, P. A. (2007). Oral Drug Therapy for Multiple Neglected Tropical Diseases: A Systematic Review. JAMA 298: 1911-1924 [Abstract] [Full text]  
• Chan, S., Bhandari, M. (2007). The Quality of Reporting of Orthopaedic Randomized Trials with Use of a Checklist for Nonpharmacological Therapies. JBJS 89: 1970-1978 [Abstract] [Full text]  
• Pildal, J, Hrobjartsson, A, Jorgensen, K., Hilden, J, Altman, D., Gotzsche, P. (2007). Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 36: 847-857 [Abstract] [Full text]  
• Turgeon, A. F., Hutton, B., Fergusson, D. A., McIntyre, L., Tinmouth, A. A., Cameron, D. W., Hebert, P. C. (2007). Meta-analysis: Intravenous Immunoglobulin in Critically Ill Adult Patients with Sepsis. ANN INTERN MED 146: 193-203 [Abstract] [Full text]  
• Bower, P., Gilbody, S., Richards, D., Fletcher, J., Sutton, A. (2006). Collaborative care for depression in primary care: Making sense of a complex intervention: systematic review and meta-regression. Br. J. Psychiatry 189: 484-493 [Abstract] [Full text]  
• Montori, V. M., Wang, Y. G., Alonso-Coello, P., Bhagra, S. (2006). Systematic Evaluation of the Quality of Randomized Controlled Trials in Diabetes. Diabetes Care 29: 1833-1838 [Abstract] [Full text]  
• Anttila, H., Malmivaara, A., Kunz, R., Autti-Ramo, I., Makela, M. (2006). Quality of reporting of randomized, controlled trials in cerebral palsy.. Pediatrics 117: 2222-2230 [Abstract] [Full text]  
• Gluud, L. L. (2006). Bias in Clinical Intervention Research. Am J Epidemiol 163: 493-501 [Abstract] [Full text]  
• Le Henanff, A., Giraudeau, B., Baron, G., Ravaud, P. (2006). Quality of Reporting of Noninferiority and Equivalence Randomized Trials. JAMA 295: 1147-1151 [Abstract] [Full text]  

Online poll

Find out more >>