Research Methods & Reporting

Rethinking pragmatic randomised controlled trials: introducing the “cohort multiple randomised controlled trial” design

BMJ 2010; 340 doi: http://dx.doi.org/10.1136/bmj.c1066 (Published 19 March 2010) Cite this as: BMJ 2010;340:c1066
  1. Clare Relton, research fellow13,
  2. David Torgerson, director of clinical trials unit2,
  3. Alicia O’Cathain, professor of health services research1,
  4. Jon Nicholl, director of the Medical Care Research Unit1
  1. 1School for Health and Related Research, University of Sheffield, Sheffield
  2. 2Department of Health Sciences, University of York, Heslington, York
  3. 3School of Healthcare, University of Leeds, Leeds
  1. Correspondence to: C Relton c.relton{at}sheffield.ac.uk
  • Accepted 16 November 2009

Pragmatic trials are important for informing routine clinical practice, but current designs have shortcomings. Clare Relton and colleagues outline the new “cohort multiple randomised controlled trial” design, which could help address the problems associated with existing approaches

Introduction

Randomised controlled trials are generally held to be the “gold standard” for establishing how well an intervention works. Trials that aim to determine the efficacy of a treatment by using a double blind, placebo controlled design (that is, explanatory trials) are, however, sometimes criticised. For example, although the design of explanatory trials results in strong internal validity—we can depend upon the results of a given trial—such trials may have limited external validity: we can’t be confident that we can apply the results to routine clinical practice. Pragmatic trials,1 2 which aim to inform healthcare decision making in practice, have been offered as a solution in that they retain the rigour of randomisation (thus eliminate selection bias) but retain the characteristics of normal clinical practice.

The implementation and interpretation of both pragmatic and explanatory randomised controlled trials are associated with significant problems. This article describes a trial design that helps address these problems—the “cohort multiple randomised controlled trial” approach.

Problems with randomised controlled trials

Existing clinical trial designs can have shortcomings in four areas: recruitment; ethics; patient preferences; and treatment comparisons.

Recruitment

The majority of randomised controlled trials have difficulty recruiting sufficient numbers of patients. For example, one investigation found that less than a third of 114 multicentre, publicly funded UK trials recruited their original target number of patients within the time originally specified.3 Failure to recruit to target may have implications for the power and generalisability of trial results.

Moreover, many clinical trials exclude hard to reach groups and ethnic minorities,4 resulting in disparities between the “with need” (reference) population and the trial population.5 6 Measures of real world effectiveness are vital for analyses of benefit, harm, and cost effectiveness. If the reference population is not adequately represented in a trial and effectiveness is variable, then such analyses cannot accurately inform real world decisions.

Ethics

The most common reason given by patients (and clinicians) for not participating in clinical trials is “concerns with information and consent.”7 In routine real world health care, patients are rarely told of treatments that their clinicians cannot with certainty provide,8 nor are patients told their treatment will be decided by chance. On the other hand, in clinical trials providing this type of “full” information before randomisation is regarded as an ethical requirement.

Patient preferences

Standard “open” (unblinded) pragmatic trials often compare an intervention with treatment as usual. Where the “standard care” on offer is available outside the trial, however, the only incentive for the patient to participate (apart from altruism) is to receive the new intervention. If a patient is allocated to treatment as usual, he or she may withdraw from the trial (attrition bias) or exhibit disappointment bias when reporting outcomes.9

Treatment comparisons

A common research scenario is addressing a clinical problem with many potential treatments. Yet often each potential treatment is trialled, one at a time, in different populations by different research teams. This approach yields many trials of different interventions, with heterogeneous trial populations and often short term and heterogeneous outcomes—a situation that is both financially and scientifically inefficient on three counts.

Firstly, lack of collection of long term outcomes hinders the measurement of infrequent adverse events and outcomes that occur far in the future. Secondly, systematic reviews of studies on a particular topic often conclude that “there was heterogeneity in populations and outcomes”; thus greater homogeneity in trial outcomes and populations is required to be able to synthesise the results of trials effectively. Thirdly, heterogeneity of trial populations and outcomes presents difficulties when making indirect comparisons between interventions; for example, the effectiveness of treatments A versus C, where only trials of treatments A versus B and B versus C exist. Indirect comparisons—where two interventions are compared through their relative effect versus a common comparator—can succeed, but sometimes result in significant discrepancies compared with the results of head to head randomised trials.10 Many competing interventions have thus not been compared, or have been compared inaccurately, which is a waste of valuable information and money.

Previous solutions

Three alternative trial designs have attempted to address the recruitment and patient preferences issues inherent in existing clinical trial designs: the patient preference,11 comprehensive cohort,12 13 and randomised consent (Zelen) designs.14

Both the patient preference design and the comprehensive cohort design make some allowance for patient preferences regarding random allocation or type of treatment by collecting data from both randomised and non-randomised patients, thus increasing the overall number of patients recruited but not the numbers randomised. Both these designs have the limitation that if large numbers of patients express a preference, there might be insufficient “indifferent” patients available to be randomised. Designs where patients are asked their preferences and randomised irrespective of these will not necessarily solve problems of attrition or failure to recruit participants with a very strong preference.15

In randomised consent (Zelen) designs, consent is sought after randomisation. However, these designs are subject to ethical criticisms, such as the lack of information regarding all trial treatment options to all patients, and scientific criticism, because of the dilution of effect owing to “cross-over” of patients to the non-randomised treatment. Despite these criticisms, reviews report the existence of more than 60 randomised consent designs with ethics committee approval.16 17 18

The “cohort multiple randomised controlled trial” design

To address some of the shortcomings of existing trial designs we propose a new approach primarily for pragmatic randomised controlled trials—the “cohort multiple randomised controlled trial” (cmRCT) design (figure).

Figure1

The “cohort multiple randomised controlled trial” design. Firstly, a large observational cohort of patients with the condition of interest is recruited (N) and their outcomes regularly measured. Then for each randomised controlled trial, information from the cohort is used to identify all eligible patients (NA). Some eligible patients (nA) are randomly selected and offered the trial intervention. The outcomes of these randomly selected patients (nA) are then compared with the outcomes of eligible patients not randomly selected; that is, those receiving usual care (NA − nA). This process can be repeated for further randomised controlled trials (for example, NB)

The key features of this design are:

  • (I) Recruitment of a large observational cohort of patients with the condition of interest

  • (II) Regular measurement of outcomes for the whole cohort

  • (III) Capacity for multiple randomised controlled trials over time

For each randomised controlled trial:

  • (IV) Identification of all eligible patients in the whole cohort (NA)

  • (V) Random selection of some patients (nA) from all eligible patients in the cohort, who are then offered the trial intervention

  • (VI) Comparison of the outcomes in randomly selected patients (nA) with the outcomes in eligible patients not randomly selected; that is, those receiving usual care (NA − nA)

  • (VII) “Patient centred” informed consent; that is, the process of obtaining patient information and consent aims to replicate that in real world routine health care.

The recruitment and regular follow-up of a large cohort of patients (features I and II) are characteristic of longitudinal observational studies. In the cmRCT design, however, all patients in the cohort consent at the outset to provide data to be used to look at the benefit of treatments for the condition of interest. Feature III, the capacity for multiple randomised controlled trials over time using patients from the same cohort, is unique to the cmRCT design. Random selection of some eligible cohort patients (feature V), the comparison of their outcomes with the outcomes in eligible patients not randomly selected (feature VI), and the similarity of the patient centred informed consent approach to real life situations (feature VII) offer solutions to the ethical criticisms of randomised consent (Zelen) designs.

Randomisation: random selection of some

Feature V and VI offer an alternative process for randomisation in clinical trials. The purpose of randomisation in experimental research is to generate two or more groups whose selection and treatment have not been influenced by anyone or anything other than chance and where all known or unknown prognostic factors are distributed evenly at baseline. Generating two groups whose membership is a result of chance can be achieved by either random allocation of all participants or random selection of some, because each approach produces the same effect. The random selection of nA patients from all patients (NA) in our example has the same effect as the random allocation of NA into two groups, nA and nB, because it is solely owing to chance whether any one patient is selected into nA. For the purposes of a randomised controlled trial, random selection from NA into nA provides two groups where all known or unknown prognostic factors are distributed at baseline purely by chance: nA and (NA − nA).

Randomisation is generally conceived as “random allocation of all” and as something that is “done” to all patients, and thus requires their prior consent. With randomisation conceived of as “random selection of some,” however, then nothing is “done” to all patients and prior consent of all patients is not required.

Information and consent: “patient centred”

The final feature of the cmRCT design (feature VII) is the adoption of a “patient centred” approach to informed consent, in which the process of obtaining patient information and consent aims to replicate that in real world routine health care rather than conform to the needs of trial design. All cohort patients consent to provide observational data at the outset; however, consent to “try” a particular intervention is sought only from those offered that intervention, thus replicating the patient centred information and consent procedures that exist in routine health care, where clinicians provide patients with the information they need, at the time they need it.

The rationale for this approach is twofold. Firstly, the primary motive for patients to enter clinical trials is not altruism, but their own direct benefit as patients.19 Clinical trial informed consent procedures should, therefore, put the needs of the patient at the centre; that is, patients should not be told about treatments that they might not then receive, nor should they be told that their treatment will be allocated by chance. Secondly, the greater the similarity between patients’ experiences in trials and their experiences in routine health care, then the greater the generalisability of the trial results to patients in routine health care.

Benefits of the approach

The cmRCT design will, we believe, help to address some of the shortcomings that prevent many pragmatic randomised controlled trials fulfilling their potential of giving robust evidence that clinicians can apply to their usual clinical populations.

Compared with randomised controlled trials, longitudinal observational studies can recruit a greater quantity and more representative sample of patients. Moreover, compared with doing individual pragmatic trials, using an observational cohort has important additional benefits:

  • A A facility for multiple randomised controlled trials

  • B Long term outcomes as standard

  • C Ongoing information as to the natural history of the condition and to treatment as usual

  • D Increased comparability between each trial conducted within the cohort

  • E Increased efficiency, particularly for expensive or high risk interventions

The cmRCT approach enables more reliable direct and indirect comparisons than is possible with trials conducted using current randomised controlled trial designs because all treatments have the same “treatment as usual” comparator and use the same core outcomes.

Furthermore, researchers using standard randomised controlled trial designs often struggle to recruit and consequently have to randomly allocate all patients to either group in equal proportions to maximise statistical power within their total sample. The large numbers of patients recruited to the cohort in the cmRCT approach increases the statistical power of any randomised controlled trials and enables unequal randomisation. For example, a small number of patients could be randomly selected to be offered an expensive treatment and compared with a larger number of unselected patients. Unequal randomisation thus improves the efficiency of trials of high cost interventions compared with equal allocation. These factors strengthen the inferences in the trial, lower treatment costs compared with standard designs (that is, once the cohort is established, it potentially allows for rapid and cheap recruitment of patients for any randomised controlled trials), and allows significant cost savings for trials of expensive treatments. Furthermore, data on treatment refusers provides information on the acceptability of the treatment and thus the generalisability of the trial results.

Role of the cmRCT design

There are certain circumstances, populations, clinical conditions, and treatments where the cmRCT design is more, or less, suitable than current strategies (box). The approach is best suited to the examination of long term conditions for which many pragmatic clinical trials will likely be conducted in the future. The design is also suitable for both primary and secondary care settings, and for conditions with easily reported patient outcomes.

Using the cohort multiple randomised controlled trial design

Most suited to:
Settings
  • Open trials with “treatment as usual” as the comparator

  • Studies that aim to inform healthcare decisions in routine practice (pragmatic trials)

  • Research questions that address easily measured and collected outcomes

Populations
  • Stable populations

  • Easily identified populations

Clinical conditions
  • Clinical conditions for which many trials will be conducted; for example, obesity, diabetes, chronic pain

  • Chronic conditions

  • Conditions for which previous trials have struggled with recruitment

Treatments
  • Treatments highly desired by patients

  • Expensive treatments

Least suited to:
Settings
  • Closed trial designs with masking or a placebo arm

  • Studies that aim to further knowledge as to how and why a treatment works (efficacy trials)

  • Research questions that address hard to measure and hard to collect outcomes

Population
  • Populations with high attrition

  • Unstable patient populations

  • Difficult to identify populations

Clinical conditions
  • Acute or short term conditions

Treatments
  • Treatments not highly desired by patients

The cmRCT design cannot be used for trials that have a placebo comparator because such trials could not use feature VI (comparison to treatment as usual) or feature VII (patient centred informed consent) as patients are never told that they may receive placebo in routine health care. However, features I to V could be used for placebo trials and benefits A to E would still accrue.

Challenges of the design

A potential problem with our approach is that significant numbers of patients may refuse to receive the intervention being trialled. An intention to treat analysis will, therefore, dilute any treatment effects. There are two ways of dealing with this problem. Firstly, we could use the statistical method complier average causal effect (CACE) analysis,20 which provides unbiased estimates of the treatment effect for patients who comply with the protocol (albeit usually with loss of power), unlike per protocol or on treatment analysis.

Secondly, we could try to avoid some potential non-compliance by presenting cohort patients with a list of possible interventions at enrolment and asking which they would consider agreeing to use if offered. This process identifies the potential compliers in advance and consequently reduces dilution effects; however, care must be taken to avoid false expectation of future treatment and the loss of feature VII, patient centred information and consent.

In researching interventions already available in routine health care, it will be necessary to identify and monitor which patients use or have used these.

Furthermore, discrete trials are currently supported by private and public funding infrastructures and institutional frameworks, to the tune of £100 000 per trial. Existing infrastructures and frameworks might struggle to determine a funding approach for cmRCTs.

Examples of the cmRCT design

Campbell and colleagues18 recently adapted the randomised consent (Zelen) method and developed an approach in which patients consented to an observational study and were then all randomly allocated to either intervention or control in a randomised controlled trial. Although this method shares several features with the cmRCT design (features I, II, IV, and VII), it does not have the capacity for multiple randomised controlled trials (feature III) or use random selection of some instead of random allocation of all (features V and VI).

We have obtained ethical approval for and have conducted a pilot study of the cmRCT design.21 In this pilot, a large observational cohort of 856 women aged 45-64 was recruited and their outcomes measured. A total of 72 women reported frequent or severe menopausal hot flushes, or both. Of these 72 women, 48 were eligible for the trial treatment (NA) and 24 were randomly selected to be offered the treatment (nA). The outcomes of the randomly selected patients were then compared with the outcomes of those eligible patients not randomly selected (NA − nA) using both intention to treat analysis and CACE analysis.20 Patients were not told about the treatments that they were not randomly selected to be offered.

The clinical outcomes of this pilot will be reported separately. However, a post hoc evaluation of the design found that the design was acceptable to patients, clinicians, and the NHS Research ethics committee. The concept of multiple trials within a single cohort of patients (feature III) has not yet been tested.

The cmRCT design is currently being used to address questions in the management of obesity (http://clahrc-sy.nihr.ac.uk/theme-obesity.html). The 20 year study is projected to recruit a cohort of 20 000 adults aged 16 years or more, and multiple trials will be embedded within this cohort. To maximise the long term benefits of this study, it is planned that the cohort will be “open” and will be replenished with new recruits (16 and 17 year olds) every two years.

Summary

The cmRCT design appears to be a workable and useful approach to pragmatic research questions that aim to inform healthcare decisions within routine practice. The design is best suited to circumstances that require open (rather than blinded) trials where “treatment as usual” is compared with the offer of study treatment, and to questions with outcomes that can be easily measured in the whole cohort (for example, patient reported outcomes). Clinical conditions where many clinical trials will be conducted and trials of desirable or expensive interventions are also well suited to the cmRCT approach.

There are challenges to the cmRCT design. Further research is required to address a range of analysis and implementation questions related to the design and the ethics of patient centred informed consent for pragmatic randomised controlled trials.

In his Harveian oration at the Royal College of Physicians, London, Professor Michael Rawlins, chair of the National Institute for Health and Clinical Excellence, called for “investigators to continue to develop and improve their methodologies in order to help decision makers appraise the evidence.”22 We hope that the cmRCT design goes some way towards addressing the problems associated with existing approaches. If these problems are addressed, then perhaps the most important problem of all will be resolved—the non-implementation of the results of clinical research.

Summary points

  • The “cohort multiple randomised controlled trial” (cmRCT) design tackles some of the problems associated with pragmatic trial designs, such as recruitment

  • The cmRCT design has several innovative features: a large observational cohort of patients is recruited and used as a multiple trials facility; each randomised controlled trial uses random selection of some participants (not random allocation of all); and “patient centred” information and consent is applied

  • The cmRCT design is best suited to: open trials where “treatment as usual” is compared with the offer of treatment; easily measured and collected outcomes; conditions where many trials will be conducted; and trials of desirable or expensive interventions

  • Further research is required to address a range of analysis, implementation, and ethical questions related to the cmRCT design

Notes

Cite this as: BMJ 2010;340:c1066

Footnotes

  • Contributors: JN and CR had the original idea for the article. This article arose from CR’s doctoral research, which was supervised by JN and AO at the School for Health and Related Research at the University of Sheffield. CR wrote the article and prepared the initial and subsequent draft. JN reviewed and commented on every draft. AO and DT commented on later drafts. AO, DT, CR, and JN agreed the final draft. CR is guarantor for the article. JN and CR jointly conceived the cmRCT design.

  • Funding: A pre-doctoral training fellowship award from the Department of Health’s National Coordinating Centre for Research Capacity Development funded CR’s doctoral research and the pilot of the cmRCT design. All work has been independent from the funders in every way.

  • Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare that (1) CR, AOC, and JN have received financial support from the University of Sheffield and DT has received financial support from the University of York for the submitted work; (2) CR, AOC, JN, and DT have no relationships with any companies that might have an interest in the submitted work in the previous three years; (3) their spouses, partners, or children have no financial relationships that may be relevant to the submitted work; and (4) no author has any non-financial interests that may be relevant to the submitted work.

  • Ethical approval: The protocol of the cmRCT pilot study was approved by the South Sheffield NHS Research Ethics Committee (ref 06/Q2305/181). NHS Scientific Review Approval was also obtained (consortium ref: ZF89).

  • Provenance and peer review: Not commissioned; externally peer reviewed.

References