- Correspondence to: D J Torgerson
- Accepted 10 July 2003
Objective To examine the prevalence of a risk of bias associated with the design and conduct of cluster randomised controlled trials among a sample of recently published studies.
Design Retrospective review of cluster randomised trials published in the BMJ, Lancet, and New England Journal of Medicine from January 1997 to October 2002.
Main outcome measures Prevalence of secure randomisation of clusters, identification of participants before randomisation (to avoid foreknowledge of allocation), differential recruitment between treatment arms, differential application of inclusion and exclusion criteria, and differential attrition.
Results Of the 36 trials identified, 24 were published in the BMJ,11 in the Lancet, and a single trial in the New England Journal of Medicine. At the cluster level, 15 (42%) trials provided evidence for secure allocation and 25 (69%) used stratified allocation. Few trials showed evidence of imbalance at the cluster level. However, some evidence of susceptibility to risk of bias at the individual level existed in 14 (39%) studies.
Conclusions Some recently published cluster randomised trials may not have taken adequate precautions to guard against threats to the internal validity of their design.
In most clinical trials participants are randomised as individuals to different treatments. Sometimes individual allocation is not possible or desirable, and groups of individuals are randomised instead: this is known as cluster or group randomisation. Many reasons for using cluster allocation exist. For example, evaluation of clinical guidelines or medical education on patient outcomes almost always requires that healthcare professionals are the “unit” of allocation.
Although randomised trials are the most robust evaluative method, poorly conducted studies are susceptible to different forms of selection bias that can make their results unsound. Methodological reviews of individually randomised trials have shown that rigorously conducted trials produce different effect estimates from poorly conducted studies.1 2 Less attention has been paid, however, to cluster trials. Cluster trials are generally more difficult to design and execute than individually randomised studies, and some design features of a cluster trial may make it especially vulnerable to a range of threats that can introduce selection bias.
In cluster trials potential bias in the execution of the trial can occur at two levels, the first of which is the cluster level. Randomisation of clusters needs to be undertaken carefully and preferably independently. Otherwise, biased allocation may occur (certain clusters being allocated to a particular arm on the basis of reasons that might affect outcome). It is theoretically possible for allocation of clusters to be subverted, as has happened in individually randomised trials.3 Similarly, once clusters have been allocated it is important, as with individually randomised trials, to try to retain the cluster in its allocated group and avoid the cluster dropping out, to avoid the risk of attrition bias.
The second level at which bias can occur in cluster trials is after the clusters have been allocated and when individual participants are recruited into the study. Sometimes identification and recruitment of participants and assessment of outcome in a cluster trial are relatively straightforward with little scope for bias. For example, in an evaluation of the effect of offering routine influenza vaccination to healthcare workers on patient mortality, hospitals were randomised to offer routine vaccination to staff or not.w1 Any differences between the groups were then observed by using mortality data. Two important methodological aspects to this trial, and other similar cluster trials, limit the risk of bias. These are complete identification and inclusion of participants, partly owing to the fact that consent was not needed for either treatment or collection of data. Because all the participants were identified and included at the point of randomisation, except for chance imbalances the two groups should be similar at baseline (assuming that the allocation procedure was fair), which avoids the threat of selection bias.
In some cluster trials identification and inclusion of participants and assessment of outcome are less straightforward. Often participants have to be recruited prospectively after randomisation. For example, in a trial of the effectiveness of a training package for general practitioners, patients had to be identified prospectively after the general practitioner had been randomised.w2 The prospective inclusion of participants can potentially lead to selection bias through the recruitment of different types of participant by the researcher or clinician. If the person prospectively recruiting participants has “foreknowledge” of the allocation group then, as shown in individually randomised trials, bias can result.3 In addition to this source of selection bias, another can be introduced by the participant if consent is needed after randomisation.
Selection bias can be introduced if consent is withheld for either treatment or data collection. This is a well known disadvantage of acquiring consent after randomisation in individually randomised trials (known as Zelen's method4), because some refusal of treatment or data collection will usually occur.5 This is less of a problem in non-Zelen designs, as participants are told in advance about the treatment options and if they decline to be exposed to one of the options they are not randomised (although some may decline in the period between allocation and receipt of treatment).
Several ways of avoiding the biases outlined above exist. One is to try to identify trial participants before randomisation and obtain consent for treatment, data collection, or both before allocation. Use of prior identification and prior consent avoids potential biases occurring through foreknowledge of the allocation schedule, by the researcher and patient. If this is not possible, identification and recruitment of participants should ideally be undertaken by someone blinded to the group allocation.
Another problem that can lead to bias, in both individual and cluster randomised trials, is the differential application of inclusion and exclusion criteria. Differential exclusion between groups in an individually randomised trial of breast cancer screening, identified in a systematic review, has led to questions about its rigour.6 Again this problem can be reduced if the person applying the criteria is blinded to the group allocation.
In this paper we review some recently published cluster trials to determine the extent of their risk of bias. We also describe the steps that some authors took to reduce these risks.
Searching and data extraction
We hand searched the BMJ, Lancet, and New England Journal of Medicine for all cluster randomised trials published from January 1997 to October 2002. We based our choice of journals on anecdotal experience that the BMJ regularly publishes cluster trials, as does the Lancet, and a wish to include a non-British general medical journal. We limited our search to five years merely so that we had a sample of fairly recent trials. We did not have a predetermined sample size.
Definition of outcomes
Selection bias can be introduced into a trial in several different ways. In this paper we sought evidence for the risk of bias from several sources.
Secure cluster allocation—This is where evidence exists that cluster randomisation was securely undertaken.
Cluster attrition—This occurs when clusters are lost to follow up after randomisation.
Cluster imbalance—This is where evidence exists of imbalance in important variables at the cluster level.
Differential individual recruitment or consent—This is when different proportions of participants are recruited to the different arms of the trial. If recruitment rates differ between groups this may lead to the risk of bias.
Differential individual exclusion or inclusion—This can occur when eligibility criteria are applied differentially after randomisation, which can introduce bias.
Two of us (SP, JW) hand searched the journals and independently extracted data. The three authors met to discuss all the papers and any disagreements. If we observed differences in proportions between the randomised groups in recruitment, consent, and exclusion or inclusion rates we used χ2 to test for significance.
We identified 42 potentially eligible trials. We excluded six studies: one was a 14 year follow up of an earlier trial,7 another measured the intervention and outcome on only one level,8 another had a switchback design,9 two guideline studies did not provide any data on individual participants,10 11 and the sixth trial had a mixture of cluster and individual allocation.12 Of the 36 trials included,w1-w36 24 were published in the BMJ, 11 in the Lancet, and one in the New England Journal of Medicine. In table 1 we describe the basic characteristics of the trials. In table 2 we examine whether the trials identified participants before random allocation and any evidence of bias occurring in the trials.
Secure cluster allocation—Fifteen trials seemed to use a secure method of allocating clusters; the remainder did not clearly describe who undertook the allocation (table 2) or how this was done. Most trials used some form of stratified random allocation to reduce the possibility of “chance bias.”
Cluster attrition—In 10 trials a loss of clusters occurred between randomisation and follow up. Most of the trials lost only a small proportion of their clusters, but one study lost more than half (56%) of all the randomised clusters.w2
Differential consent or recruitment—We found some evidence for differential consent or recruitment in seven of the 23 trials that had not undertaken prior identification of participants (table 2). Three trials recruited more participants from one group than the other,w11 w16 w31 and the other four studies differentially obtained consent from more participants in one arm than the other.w9 w13 w29 w34 One trial, although it seemed to identify all the participants before allocation for the main mortality outcome, seemed to have introduced the risk of selection bias into the measurement of its secondary outcome.w1
Differential application of inclusion or exclusion criteria—We found two trials that seemed to have applied inclusion or exclusion criteria differentially between groups after randomisation.w20 w25 Moher et al, in a study promoting methods of secondary prevention of coronary disease, excluded significantly more participants owing to misdiagnosis in the intervention groups than in the control group.w20 Similarly, Olivarius et al excluded twice as many participants because of illness in the intervention group than in the control arm.w25
Differential attrition—Evidence of differential attrition between the randomised groups existed in four trials.w8 w18 w27 w36
Table 3 summarises the potential sources of bias risk in 14 trials in which we observed differences between the groups that indicate a risk of selection bias. Authors of six studies alerted the reader to the potential risk of bias in their study.
Cluster trials can be difficult to do; nevertheless, they are needed to evaluate some interventions. Although a large literature exists about sources of potential bias that can occur in individually randomised trials, less evidence is available about the special problems encountered in cluster trials.
Evidence of bias risk at cluster level
Some authors did not clearly describe the allocation process of the clusters, which is important as this can be subverted; other trialists were clear in stating that an independent person undertook the allocation. In most trials some form of stratification was used to reduce the element of chance bias, although this was not always successful. Some trials lost complete clusters after randomisation. However, with the exception of one trial,w2 the proportion of clusters lost was relatively low and therefore would be unlikely to introduce bias.
Evidence of bias risk at individual level
One of the major risks for introduction of bias is when prospective recruitment is needed. This difficulty can be overcome and the risk of bias reduced, as two examples serve to illustrate. Bennewith et al reduced the possibility of recruitment bias by blinding the clinician identifying participants until after the patient was assessed as being eligible or not.w4 Similarly, King et al reduced the same threat by asking a trained receptionist to recruit patients.w2 Because that trial evaluated a training package to help general practitioners to manage depressed patients, the training would probably have reduced the diagnostic threshold of the general practitioners. Thus, had the doctors recruited participants themselves, this would have increased the risk that they could have recruited either more or less seriously depressed participants than the control doctors. Use of receptionists reduced this risk.
As well as differences between groups in recruitment and consent, we found that differential post-randomisation exclusion or inclusion was a problem in some studies. Inclusion of the “wrong” participant is likely to be a problem in some cluster trials. Two ways exist to deal with wrong inclusions and avoid bias. Firstly, all participants could be retained within the trial after allocation whether or not they fitted the inclusion criteria and even if they could not or did not receive the allocated treatment (that is, intention to treat analysis).13 This could lead, however, to some dilution of treatment effect. As an alternative, decisions on exclusion could be made by a person blind to the group allocation.
A new CONSORT statement?
Elbourne and Campbell have recently argued for amending the CONSORT statement to allow for the special methodological circumstances of cluster trials.14 We would echo this call. We found it very difficult in several of the trials to ascertain whether a risk of bias was likely or not. We would wish the following additions to be made. Firstly, a clear statement as to whether the population was identified before or after the allocation decision had been made. Secondly, was the person who recruited the participants blind to group allocation? Thirdly, what was the size of the population within the clusters? For example, Steptoe et alw31 did not state the size of the general practice populations in their trial arms and Kinmonth et alw15 gave only means. For the first of these studies we could only assume that the recruitment was significantly different, and for the second study we had to make an estimate. This missing information also meant that for some studies we could not be completely sure if recruitment bias had taken place. For example, in Kendrick et al no suggestion of any recruitment bias was apparent; however, we could not be absolutely sure as the authors did not present the practice population sizes.w14
Cluster trials are vulnerable to the risk of bias. Careful planning and execution of such trials can avoid these biases.
What is already known on this topic
Reviews of individually randomised trials show that results can differ according to quality of methods
Foreknowledge of allocation and failure to use intention to treat analysis can lead to bias
What this study adds
Cluster randomised trials are susceptible to forms of selection bias
Careful planning and execution of such trials can avoid these biases
Additional references appear on bmj.com
We thank Doug Altman, Diana Elbourne, Craig Ramsay, and Trevor Sheldon for their helpful comments on an earlier version of this manuscript.
Contributors DJT suggested the idea of the review and wrote the first draft. SP and JW did the searches and extracted data from the included papers; they also helped to write and comment on the draft manuscript. DJT is the guarantor.
Funding JW is trial coordinator of the SAPPHIRE trial funded by the Medical Research Council; SP and DJT are funded by the University of York.
Competing interests None declared.
Ethical approval Not needed.