- Gunn Elisabeth Vist, senior researcher (gev{at}nhsrc.no)1,
- Kâre Birger Hagen, senior researcher2,
- P J Devereaux, senior researcher3,
- Dianne Bryant, senior researcher3,
- Doris Tove Kristoffersen, statistician1,
- Andrew David Oxman, researcher1
- 1Norwegian Health Services Research Centre, PO Box 7004, 0130 Oslo, Norway
- 2National Research Centre for Rehabilitation in Rheumatology, Diakonhjemmet Hospital, Oslo
- 3Clinical Epidemiology and Biostatistics, McMaster University Health Sciences Centre, Hamilton, Ontario, Canada
- Correspondence to: G E Vist
- Accepted 31 March 2005
Abstract
Objective To systematically compare the outcomes of participants in randomised controlled trials (RCTs) with those in comparable non-participants who received the same or similar treatment.
Summary of studies included in sensitivity analysis
Data sources Bibliographic databases, reference lists from eligible articles, medical journals, and study authors.
Review methods RCTs and cohort studies that evaluated the clinical outcomes of participants in RCTs and comparable non-participants who received the same or similar treatment.
Results Five RCTs (six comparisons) and 50 cohort studies (85 comparisons) provided data on 31 140 patients treated in RCTs and 20 380 comparable patients treated outside RCTs. In the five RCTs, in which patients were given the option of participating or not, the comparisons provided limited information because of small sample sizes (a total of 412 patients) and the nature of the questions considered. 73 dichotomous outcomes were compared, of which 59 reported no statistically significant differences. For patients treated within RCTs, 10 comparisons reported significantly better outcomes and four reported significantly worse outcomes. Significantly heterogeneity was found (I2 = 89%) among the comparisons of 73 dichotomous outcomes; none of our a priori explanatory factors helped explain this heterogeneity. The 18 comparisons of continuous outcomes showed no significant differences in heterogeneity (I2 = 0%). The overall pooled estimate for continuous outcomes of the effect of participating in an RCT was not significant (standardised mean difference 0.01, 95% confidence interval −0.10 to 0.12).
Conclusion No strong evidence was found of a harmful or beneficial effect of participating in RCTs compared with receiving the same or similar treatment outside such trials.
Introduction
Properly conducted randomised controlled trials (RCTs) provide the strongest evidence of the effects of treatment.1 It is, however, controversial as to whether participants of such trials benefit directly or whether these studies are solely for the benefit of future patients. In addition, there is much scepticism about the applicability of the results to usual practice.2
Four reviews that considered whether it is beneficial or harmful to participate in RCTs drew varied conclusions.3–6 These reviews compared patients who were treated within trials with those treated outside the trials, regardless of differences between the clinical interventions or between the participants and non-participants. It is therefore uncertain whether the results reflect the effects of participating in an RCT (trial effects), differences in the clinical interventions (treatment effects), or differences between participants and non-participants. We determined whether the outcomes of participants in RCTs differed from those of comparable non-participants who received the same or similar treatment.
Methods
Our review was undertaken as a Cochrane methodology review (see Cochrane Library for fuller details of our methods and updated versions of the review).7
We included studies that compared participants in RCTs with comparable non-participants who received the same or similar treatment. We included observational studies and RCTs in which participation or the option of participation was randomly allocated.
Search strategies
We used seven strategies to identify relevant studies: consultation with experts; search of personal files; electronic searches of the Cochrane central register of controlled trials, Medline, Embase, the Cochrane methodology register, and PsycINFO; a review of reference lists from eligible articles; and a search of PubMed using the “related articles” feature and SciSearch. We also hand searched articles published in 2000 in five medical journals (BMJ, Annals of Internal Medicine, JAMA, Lancet, and New England Journal of Medicine) to identify RCTs with over 200 patients and at least 100 eligible non-participants. Studies' authors were contacted for data on the treatment and outcomes of eligible non-participants.
Assessment of study eligibility
Two reviewers independently assessed each article for eligibility. Disagreements were resolved by discussion. A third reviewer was consulted when consensus could not be reached.
Data abstraction
Two reviewers independently abstracted data from eligible studies. Each study was assessed for selection bias (differences between participants and non-participants), detection bias, and exclusion bias (losses to follow-up). On the basis of the combined risks of the three biases, we grouped each comparison into overall quality groups (randomised, controlled comparisons, partially controlled comparisons, and poorly controlled comparisons) for analysis. Missing data were sought from the investigators.
Analysis
We compared the experimental group of the RCT with their respective eligible non-participants who received the treatment and the control group and eligible non-participants who received the control treatment outside of the trial. For each comparison, we analysed the main outcome, and, when reported, we analysed mortality separately. For all of the included comparisons we used the main outcome as reported by the investigators. We analysed the dichotomous and continuous results separately. The results are reported as relative risks with 95% confidence intervals, using adjusted estimates when available. For a summary of the dichotomous results in one table, we calculated the associated standard error from the natural logarithm of the unadjusted relative risk. Heterogeneity was assessed by χ2 test and the I2 statistic using RevMan version 4.2.8 For the unadjusted relative risk analysis, we used the Mantel-Haenszel test in RevMan. A fixed effect model was used to calculate summary statistics if no statistically significant (P < 0.10) heterogeneity was found among similar comparisons. For statistically heterogeneous results we described the variation in the estimates and key explanatory factors; where possible relating the explanatory factors to observed differences in estimates of the effects of participation. We constructed a funnel plot to explore the possibility of publication bias.
Results
Overall, 55 studies, totalling 91 comparisons, met our inclusion criteria (fig 1). Forty one studies are still awaiting assessment, which currently cannot be included or excluded on the basis of the published information.
We identified five RCTs (six comparisons) in which patients were randomised according to whether they had the option to participate. These studies provided limited data because of their small sample sizes and the nature of the questions considered. Two studies randomised 82 patients to “n of 1 trials” compared with standard practice—that is, randomised, double blind, multiple crossover comparisons of an active drug with a placebo in a single patient.9 10 One study (60 patients) measured spontaneously self reported side effects in patients who had or had not been informed that they were in an RCT.11 One study (227 patients) reported satisfaction among patients randomised to an RCT compared with patients randomised to a patient preference trial in which they had a choice of treatment.12 Another study (43 patients) reported pain reduction among patients randomised to an RCT compared with those who were not invited to participate.13 None of these studies found significant differences in outcomes between patients treated in or outside RCTs.
Non-randomised cohort studies
We identified 50 cohort studies (85 comparisons) totalling 30 862 patients participating in RCTs compared with 20 246 patients treated outside RCTs.14–13 Seventy comparisons comprised dichotomous outcomes, of which 12 reported adjusted estimates, and 15 comparisons comprised continuous outcomes.
We found significant heterogeneity (I2 = 89.0%) among the results of comparisons with dichotomous main outcomes (fig 2); these results were therefore not pooled. Of these 73 comparisons, 59 reported no significant differences between outcomes for patients treated in RCTs and those receiving similar treatments outside RCTs; 10 reported significantly better outcomes for patients treated in RCTs, and four reported significantly worse outcomes for patients treated in RCTs.
Results of dichotomous main outcomes in participants of randomised controlled trials and comparable non-participants who received the same or similar treatment
Figure 3 shows the results of the 18 comparisons with continuous main outcomes. We found no significant heterogeneity (I2 = 0%). The pooled estimate found no differences in outcomes for patients treated in and outside RCTs (standardised mean difference 0.01, 95% confidence interval −0.10 to 0.12).
Results of continuous main outcomes in participants of randomised controlled trials and comparable non-participants who received the same or similar treatment
In 17 studies (32 comparisons) with data on mortality (fig 4), we found significant heterogeneity (I2 = 88.8%); the results were therefore not pooled. In 24 of the 32 comparisons we found no significant difference in mortality. Four comparisons reported a significant lower risk of mortality for patients treated in RCTs and four comparisons reported a significantly higher risk of mortality.
Comparison of mortality between participants of randomised controlled trials and comparable non-participants who received the same or similar treatment
Separate subgroup analyses could not explain the observed heterogeneity by the different types of eligible non-participants, treatments, clinical specialities, or study quality (selection bias, detection bias, and exclusion bias). The table summarises the studies included in the sensitivity analysis.
The funnel plot of the dichotomous comparisons showed no asymmetry (fig 5), indicating a low risk of publication bias.
Funnel plot of dichotomous comparisons in participants of randomised controlled trials and comparable non-participants who received the same or similar treatment
Discussion
Our systematic review found no strong evidence of a harmful or beneficial trial effect of participating in randomised controlled trials (RCTs). The five included RCTs provided limited evidence because of their small sample sizes and the nature of the questions they considered, but they did show that it is possible to consider questions about the effects of participating in RCTs by using randomised designs. Our interpretation of the 50 non-randomised cohort studies was limited by the quality and size of the comparisons and the wide variations in participants, clinical interventions, and outcomes. Most of the 85 non-randomised cohort comparisons found no statistically significant differences, although 10 reported better outcomes for patients in RCTs and four reported better outcomes for patients outside of RCTs.
Previous reviews that considered a less precise question than the one we evaluated drew varied conclusions. For example, one identified 14 articles reporting data from 21 trials and concluded that, if anything, randomised trials tend to have beneficial rather than harmful effects.4 Another review included seven of these 14 articles and 17 additional articles.6 Only eight of the studies compared trial patients with non-trial patients who met the same eligibility criteria, and it was only possible to separate treatment effects from trial effects in three of these trials. A further review found 10 comparisons on survival or quality of life of patients treated in RCTs for life threatening illnesses (eight were cancer treatments) with those treated outside RCTs.5 It found evidence of longer survival in participants, but the authors were not confident of the results.
Our review differs from these reviews in several ways, including the scope and comprehensiveness of our search, our method of analysis, and the question we asked, which controlled for differences in the effects of different interventions and differences between participants and non-participants. (See bmj.com for an evidence profile of our results according to the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) Working Group.) Our results are based mainly on comparisons of cohorts and are subject to the usual uncertainty associated with observational studies.1 Additionally, we could not explain the significant heterogeneity between studies, which affects our confidence in the results and reduces the overall quality of information. Other relevant studies apart from those included in this review may exist, as indicated by the number of studies awaiting assessment and the difficulty we and others had in searching for this type of study in electronic databases. As we did not find evidence of publication bias, it is unlikely that the studies that we failed to identify would provide strong evidence of either harmful or beneficial effects.
An important corollary of this finding is that it counters suggestions that the results of RCTs cannot be applied to usual clinical practice, because most of the studies found no significant difference in outcomes for participants of RCTs compared with comparable non-participants who received similar treatment.
In most cases, RCTs seem to provide estimates of treatment effects that are applicable to comparable patients who receive similar interventions in usual clinical practice. In addition to being informed about the risks and harms of an intervention when invited to participate in RCTs, patients can be told that, independently of the effects of the interventions being compared, participating in a trial is likely to result in similar outcomes to patients who receive the same or similar treatment outside of the trial.
What is already known on this topic
Some people believe that participation in a randomised controlled trial (RCT) increases a patient's risk of a bad outcome
Some people claim that the results of RCTs are not applicable to usual clinical practice
What this study adds
Participants in RCTs had similar outcomes to comparable patients who received the same or similar treatment outside the trial
The results of RCTs are therefore applicable to comparable patients in usual clinical practice
A giving the evidence profile of results is on bmj.com table
Acknowledgments
We thank Dave Sackett for initiating this review, for his contributions to the protocol, and for his support; Lena Nordheim for help with the search strategy; and the following for additional information about their studies: S M Albert, B J Barrett, J M Bennett, S B Black, S S Brehm, G Berglund, J F Bergmann, N Bijker, M Blichert-Toft, L Boros, J G Cairncross, K M Carroll, D W Chadwick, S P Chauhan, C Chilvers, G P Clagett, C J Cohen, K Collins, J S Cooper, K G Cooper, V Cottin, U Creutzig, S Davis, E P DiMagno, J van Eys, P J Ferrone, G M Forbes, J Frisell, B Glimelius, C Gluud, F Guillemin, J C Hauth, M Helsing, D Hellberg, J M Henderson, A M Hutter, L P Jensen, J J C Jonker, T Karrison, S B King, D Kjar, R A Kronmal, R W Licht, E Lidbrink, W F Liu, J K Madsen, J L Mahon, M A Malangoni, C Marcocci, J R McKay, M P Mourits, C Moynihan, G R Mundy, C D Naylor, M G Neuwirth, J R O'Fallon, J L Paradise, A Pinchera, M J Playforth, A V Pollock, P D Rokke, M A Rosen, M M Rovers, R G Rychtarik, B Schmidt, C Schmoor, B E Sha, J H Shelhamer, B G Stegmayr, C A Stiller, J M Stone, R Takolander, I F Tannock, P Vestergaard, B Ward, C Weijer, D J Weisdorf, D G Wyse, B Yersin, and V L Yu.
Footnotes
-
Contributors GEV carried out the literature search. GEV and KBH selected the studies. GEV, KBH, PJD, and DB extracted the data. GEV, DTK, and ADO analysed the data. GEV, KBH, PJD, DB, and ADO drafted the manuscript. GEV, KBH, PJD, and ADO developed the protocol and review. DB hand searched the five medical journals. GEV is guarantor for the paper.
-
Funding Norwegian Health Services Research Centre, McMaster University, and the Nuffield Trust.
-
Competing interests None declared.
-
Ethical approval Not required.