BMJ  2005;330:1175 (21 May), doi:10.1136/bmj.330.7501.1175

Paper

Systematic review to determine whether participation in a trial influences outcome

¹ Norwegian Health Services Research Centre, PO Box 7004, 0130 Oslo, Norway, ² National Research Centre for Rehabilitation in Rheumatology, Diakonhjemmet Hospital, Oslo, ³ Clinical Epidemiology and Biostatistics, McMaster University Health Sciences Centre, Hamilton, Ontario, Canada

Abstract

Objective To systematically compare the outcomes of participants in randomised controlled trials (RCTs) with those in comparable non-participants who received the same or similar treatment.

Data sources Bibliographic databases, reference lists from eligible articles, medical journals, and study authors.

Review methods RCTs and cohort studies that evaluated the clinical outcomes of participants and comparable non-participants who received the same or similar treatment.

Results Five RCTs (six comparisons) and 50 cohort studies (85 comparisons) provided data on 31 140 patients treated in RCTs and 20 380 comparable patients treated outside RCTs. In the five RCTs, in which patients were given the option of participating or not, the comparisons provided limited information because of small sample sizes (a total of 412 patients) and the nature of the questions considered. 73 dichotomous outcomes were compared, of which 59 reported no statistically significant differences. For patients treated within RCTs, 10 comparisons reported significantly better outcomes and four reported significantly worse outcomes. Significantly heterogeneity was found (I² = 89%) among the comparisons of 73 dichotomous outcomes; none of our a priori explanatory factors helped explain this heterogeneity. The 18 comparisons of continuous outcomes showed no significant differences in heterogeneity (I² = 0%). The overall pooled estimate for continuous outcomes of the effect of participating in an RCT was not significant (standardised mean difference 0.01, 95% confidence interval -0.10 to 0.12).

Conclusion Participating in RCTs neither harms nor benefits participants, compared with receiving the same or similar treatment outside such trials.

Introduction

Few reviews have considered whether it is beneficial or harmful to participate in randomised controlled trials (RCTs).^1-4 Therefore it remains unclear whether participants in randomised controlled trials benefit directly or whether such studies are solely for the benefit of future patients. Much scepticism also surrounds the applicability of the results to usual practice.⁵

We determined whether the outcomes of participants in RCTs differed from those of comparable non-participants who received the same or similar treatment.

Methods

We included studies that compared participants in RCTs with comparable non-participants who received the same or similar treatment. Our study included observational studies and RCTs in which participation or the option of participation was randomly allocated.

We searched PubMed, the Cochrane central register of controlled trials, Medline, Embase, the Cochrane methodology register, and PsycINFO; our personal files; and the reference lists of relevant articles, and we hand searched five medical journals. We contacted the studies authors for data on eligible non-participants, and also contacted experts.

Two reviewers independently assessed each article for eligibility and two reviewers independently abstracted data from relevant studies. Studies were assessed for selection bias, detection bias, and exclusion bias. On the basis of the combined risks of the three biases, we grouped each comparison into quality groups for analysis.

We compared the experimental group of the RCT with their respective eligible non-participants who received the treatment and the control group and eligible non-participants who received the control treatment. For each comparison, we analysed the main outcome; when mortality was reported, we analysed it separately. We analysed the dichotomous and continuous results separately. The results are reported as relative risks with 95% confidence intervals, with adjusted estimates when available. Heterogeneity was assessed by ² test and the I² statistic.⁶ For the unadjusted relative risk analysis, we used the Mantel-Haenszel test. A fixed effect model was used to calculate summary statistics if no statistically significant (P < 0.10) heterogeneity was found among similar comparisons. We constructed a funnel plot to explore the possibility of publication bias.

View larger version (55K): [in this window] [in a new window]   Fig 1 Results of dichotomous main outcomes in participants of randomised controlled trials and comparable non-participants who received the same or similar treatment

 

Results

Overall, 55 studies, totalling 91 comparisons, met our inclusion criteria (see bmj.com). Forty one studies are still awaiting assessment.

View larger version (63K): [in this window] [in a new window]   Fig 2 Results of continuous main outcomes in participants of randomised controlled trials and comparable non-participants who received the same or similar treatment

 
We identified five RCTs (six comparisons) in which patients were randomised according to whether they had the option to participate. These studies provided limited data because of their small sample sizes and the nature of the questions considered. Two studies randomised 82 patients to "n of 1 trials" compared with standard practice.^{7 8} One study (60 patients) measured spontaneously self reported side effects in patients who had or had not been informed that they were in an RCT.⁹ One study (227 patients) reported satisfaction among patients randomised to an RCT compared with patients randomised to a patient preference trial in which they had a choice of treatment.¹⁰ Another study (43 patients) reported pain reduction among patients randomised to an RCT compared with those who were not invited to participate.¹¹ None of these studies found significant differences in outcomes between patients treated in or outside RCTs.

We identified 50 cohort studies (85 comparisons) totalling 30 862 patients participating in RCTs compared with 20 246 patients treated outside RCTs.^w1-w50 Seventy comparisons comprised dichotomous outcomes, of which 12 reported adjusted estimates, and 15 comparisons comprised continuous outcomes.

We found significant heterogeneity (I² = 89.0%) among the results of comparisons with dichotomous main outcomes (fig 1). Of these 73 comparisons, 59 reported no significant differences between outcomes for patients treated in RCTs and those receiving similar treatments outside RCTs, 10 reported significantly better outcomes for patients treated in RCTs, and four reported significantly worse outcomes for patients treated in RCTs.

Figure 2 shows the results of the 18 comparisons with continuous main outcomes. We found no significant heterogeneity (I² = 0%). The pooled estimate found no differences in outcomes for patients treated in and outside RCTs (standardised mean difference 0.01, 95% confidence interval -0.10 to 0.12).

In 17 studies (32 comparisons) with data on mortality (fig 3), we found significant heterogeneity (I² = 88.8%). In 24 of the 32 comparisons we found no significant difference in mortality. Four comparisons reported a significant lower risk of mortality for patients treated in RCTs and four comparisons reported a significantly higher risk of mortality.

View larger version (52K): [in this window] [in a new window]   Fig 3 Comparison of mortality between participants of randomised controlled trials and comparable non-participants who received the same or similar treatment

 

Separate subgroup analyses could not explain the observed heterogeneity by the different types of eligible non-participants, treatments, clinical specialities, or study quality (selection bias, detection bias, and exclusion bias). See bmj.com for a summary of the studies included in the sensitivity analysis.

A funnel plot of the dichotomous comparisons showed no asymmetry (see bmj.com), indicating a low risk of publication bias.

Discussion

Our systematic review found no strong evidence of a harmful or beneficial trial effect of participating in randomised controlled trials (RCTs). The five included RCTs provided limited evidence because of their small sample size and the nature of the question considered, but showed that it is possible to consider questions about the effects of participating in RCTs by using randomised designs. Our interpretation of the 50 non-randomised cohort studies was limited by the quality and size of the comparisons and the wide variations in participants, clinical interventions, and outcomes. Most of the 85 non-randomised cohort comparisons found no statistically significant differences, although 10 reported better outcomes for patients in RCTs and four reported better outcomes for patients outside of RCTs.

Previous reviews that considered a less precise question than the one we evaluated drew varied conclusions. Our review differs from these reviews in several ways, including the scope and comprehensiveness of our search, our method of analysis, and the question we asked, which controlled for differences in the effects of different interventions and differences between participants and non-participants. Our results are based mainly on comparisons of cohorts and are subject to the usual uncertainty associated with observational studies.¹² Additionally, we could not explain the significant heterogeneity between studies. Other relevant studies apart from those included in this review may exist, as indicated by the number of studies awaiting assessment and the difficulty encountered in searching for this type of study. As we did not find evidence of publication bias, and it is unlikely that the studies that we failed to identify would provide strong evidence of either harmful or beneficial effects.

An important corollary of this finding is that it counters suggestions that the results of RCTs cannot be applied to usual clinical practice, because most of the studies found no significant difference in outcomes for participants of RCTs compared with comparable non-participants who received similar treatment.

What is already known on this topic Some people believe that participation in a randomised controlled trial (RCT) increases a patient's risk of a bad outcome Some people claim that the results of RCTs are not applicable to usual clinical practice What this study adds Participants in RCTs had similar outcomes to comparable patients who received the same or similar treatment outside the trial The results of RCTs are therefore applicable to comparable patients in usual clinical practice

---

A table giving the evidence profile of results is on bmj.com

This is the abridged version; the full version is on bmj.com

Editorial by Sackett

See bmj.com for acknowledgments

Contributors: See bmj.com

Funding: Norwegian Health Services Research Centre, McMaster University, and the Nuffield Trust.

Competing interests: None declared.

Ethical approval: Not required.

References

1. Stiller CA. Centralised treatment, entry to trials and survival. Br J Cancer 1994;70: 352-62.[Web of Science][Medline]
2. Braunholtz DA, Edwards SJL, Lilford RJ. Are randomized clinical trials good for us (in the short term)? Evidence for a "trial effect." J Clin Epidemiol 2001;54: 217-24.[CrossRef][Web of Science][Medline]
3. Emergency Care Research Institute 2002. Patients' reasons for participation in clinical trials and effect of trial participation on patient outcomes. www.ecri.org/Patient_Information/Patient_Reference_Guide/evidence.pdf (accessed Nov 2004).
4. Peppercorn JM, Weeks JC, Cook EFC, Joffe S. Comparison of outcomes in cancer patients treated within and outside clinical trials: conceptual framework and structured review. Lancet 2004;363: 263-70.[CrossRef][Web of Science][Medline]
5. Rothwell PM. External validity of randomised controlled trials: "To whom do the results of this trial apply?" Lancet 2005;365: 82-93.[CrossRef][Web of Science][Medline]
6. Higgins JP, Thompson SG. Quantifying heterogeneity in meta-analysis. Stat Med 2002;21: 1539-58.[CrossRef][Web of Science][Medline]
7. Mahon J, Laupacis A, Donner A, Wood T. Randomised study of n of 1 trials versus standard practice. BMJ 1996;312: 1069-74.[Abstract/Free Full Text]
8. Mahon JL, Laupacis A, Hodder RV, McKim DA, Paterson NAM, Wood TE, et al. Theophylline for irreversible chronic airflow limitation. A randomized study comparing n of 1 trials to standard practice. Chest 1999;115: 38-48.[Abstract/Free Full Text]
9. Dahan R, Caulin C, Figea L, Kanis JA, Caulin F, Segrestaa JM. Does informed consent influence therapeutic outcome? A clinical trial of the hypnotic activity of placebo in patients admitted to hospital. BMJ 1986;293: 363-4.
10. Cooper KG, Grant AM, Garratt AM. The impact of using a partially randomised patient preference design when evaluating alternative managements for heavy menstrual bleeding. Br J Obstet Gynaecol 1997;104: 1367-73.[Web of Science][Medline]
11. Bergmann JF, Chassany O, Gandiol J, Deblois P, Kanis JA, Segrestaa JM, et al. A randomised clinical trial of the effect of informed consent on the analgesic activity of placebo and naproxen in cancer pain. Clin Trials Metaanal 1994;29: 41-7.[Medline]
12. Kunz R, Vist G, Oxman AD. Randomisation to protect against selection bias in healthcare trials (Cochrane methodology review). In: Cochrane Library. Issue 4. Oxford: Update Software, 2002.

(Accepted 31 March 2005)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Articles

Trial outcomes are applicable to all comparable patients
BMJ 2005 330: 0. [Full Text]

Simple problems please, and one at a time

Fiona Godlee
BMJ 2005 330: 0. [Extract] [Full Text] [PDF]

Participants in research

David L Sackett
BMJ 2005 330: 1164. [Extract] [Full Text] [PDF]

Antidepressant drugs and generic counselling for treatment of major depression in primary care: randomised trial with patient preference arms

Clair Chilvers, Michael Dewey, Katherine Fielding, Virginia Gretton, Paul Miller, Ben Palmer, David Weller, Richard Churchill, Idris Williams, Navjot Bedi, Conor Duggan, Alan Lee, and Glynn Harrison
BMJ 2001 322: 772. [Abstract] [Full Text] [PDF]

Randomised study of n of 1 trials versus standard practice

Jeffrey Mahon, Andreas Laupacis, Allan Donner, and Thomas Wood
BMJ 1996 312: 1069-1074. [Abstract] [Full Text]

This article has been cited by other articles:

• Clark, A. L., Lammiman, M. J., Goode, K., Cleland, J. G.F. (2009). Is taking part in clinical trials good for your health? A cohort study. Eur J Heart Fail 11: 1078-1083 [Abstract] [Full text]  
• Chalkidou, K., Lord, J., Fischer, A., Littlejohns, P. (2008). Evidence-Based Decision Making: When Should We Wait For More Information?. Health Aff (Millwood) 27: 1642-1653 [Abstract] [Full text]  
• McAlister, F. A. MD MSc (2008). The "number needed to treat" turns 20 -- and continues to be used and misused. CMAJ 179: 549-553 [Full text]  
• Julian-Reynier, C., Geneve, J., Dalenc, F., Genre, D., Monnier, A., Kerbrat, P., Largillier, R., Serin, D., Rios, M., Roche, H., Jimenez, M., Tarpin, C., Maraninchi, D. (2007). Assessment of Care by Breast Cancer Patients Participating or Not Participating in a Randomized Controlled Trial: A Report With the Patients' Committee for Clinical Trials of the Ligue Nationale Contre le Cancer. JCO 25: 3038-3044 [Abstract] [Full text]  
• Steg, P. G., Lopez-Sendon, J., Lopez de Sa, E., Goodman, S. G., Gore, J. M., Anderson, F. A. Jr, Himbert, D., Allegrone, J., Van de Werf, F., for the GRACE Investigators, (2007). External Validity of Clinical Trials in Acute Myocardial Infarction. Arch Intern Med 167: 68-73 [Abstract] [Full text]  
• Hassett, M. J., O'Malley, A. J., Pakes, J. R., Newhouse, J. P., Earle, C. C. (2006). Frequency and cost of chemotherapy-related serious adverse effects in a population sample of women with breast cancer.. JNCI J Natl Cancer Inst 98: 1108-1117 [Abstract] [Full text]  
• Mills, E. J., Nachega, J. B., Buchan, I., Orbinski, J., Attaran, A., Singh, S., Rachlis, B., Wu, P., Cooper, C., Thabane, L., Wilson, K., Guyatt, G. H., Bangsberg, D. R. (2006). Adherence to antiretroviral therapy in sub-Saharan Africa and North America: a meta-analysis.. JAMA 296: 679-690 [Abstract] [Full text]  
• Sackett, D. L (2005). Participants in research. BMJ 330: 1164-1164 [Full text]  

Rapid Responses:

Read all Rapid Responses

Extrapolation of clinical trials is not as straightforward as it might seem at first glance

Zvonko Rumboldt, et al.
bmj.com, 21 May 2005 [Full text]

Confidence intervals not correctly reported

Claudia Schmoor, et al.
bmj.com, 29 May 2005 [Full text]

Confidence intervals for adjusted results

Gunn Elisabeth Vist, et al.
bmj.com, 31 May 2005 [Full text]

Online poll

Find out more here>> and here>>