Education and debate

Interpreting the evidence: choosing between randomised and non-randomised studies

Martin McKee, professor of European public health ^a,  Annie Britton, research fellow ^a,  Nick Black, professor of health services research ^a,  Klim McPherson, professor of public health epidemiology ^a,  Colin Sanderson, senior lecturer in health services research ^a,  Chris Bain, reader in social and preventive medicine ^b. 

^a London School of Hygiene and Tropical Medicine, London WC1E 7HT, ^b University of Queensland Medical School, Brisbane 4006, Australia

Correspondence to: Martin McKeem.mckee{at}lshtm.ac.uk

Evaluations of healthcare interventions can either randomise subjects to comparison groups, or not. In both designs there are potential threats to validity, which can be external (the extent to which they are generalisable to all potential recipients) or internal (whether differences in observed effects can be attributed to differences in the intervention). Randomisation should ensure that comparison groups of sufficient size differ only in their exposure to the intervention concerned. However, some investigators have argued that randomised controlled trials (RCTs) tend to exclude, consciously or otherwise, some types of patient to whom results will subsequently be applied. Furthermore, in unblinded trials the outcome of treatment may be influenced by practitioners' and patients' preferences for one or other intervention. Though non-randomised studies are less selective in terms of recruitment, they are subject to selection bias in allocation if treatment is related to initial prognosis.

These issues have led to extensive debate, although empirical evidence is limited. This paper is a brief summary of a more detailed review¹ of the impact of these potential threats.

	Nature of the evidence

The review focused on threats to internal and external validity of evaluations of effectiveness and on the strategies proposed to overcome them (table). Various factors act through their effect on the distribution of the potential to benefit among different groups. This can be illustrated schematically (fig 1). The reference population is defined by an envelope, represented here as a triangle but potentially taking many shapes. At some point, a threshold is reached, below which the overall risks outweigh the benefits. As patients are excluded or do not participate, the study population becomes a progressively smaller subset of the reference population, in principle increasing the scope for selection bias and raising the question of whether it is valid to apply the results obtained to the reference population.

View larger version (37K): [in this window] [in a new window]   Fig 1.    Differences in inclusion and participation. Shaded areas represent the study population

We used systematic reviews to explore the potential and actual importance of factors that lead to selective recruitment, examining four questions:

• Do non-randomised studies give systematically different measurements of treatment effect from RCTs?
• Are there systematic differences between the subjects included in or excluded from studies, and do these influence the measured treatment effect?
• To what extent is it possible to overcome known or unknown baseline differences between groups that are not allocated randomly?
• How important are patients' preferences for an intervention and, if patients are randomised to a treatment they would not choose, how does this affect their outcome?

	Findings

Comparing results of RCTs and non-randomised studies
Eighteen papers were identified where a single intervention was evaluated by both methods (a full list is available on the BMJ's website). A review was published just after our original report; on the basis of eight comparisons it found that, on average, non-randomised studies overestimate effect size.² In contrast, of the seven studies in our review where the two methods detected effects in the same direction, in three the effect size was greater in the randomised trial and in four it was greater in the non-randomised study. The key finding in our study is that neither method consistently gave larger estimates of treatment effect.

• In non-randomised studies, patients tend to be allocated to treatments that are correctly considered most appropriate for their individual circumstances;
• Exclusions from a RCT create a sample with less capacity to benefit than in a non-randomised study;
• An unblinded RCT fails to capture patients with strong preferences for a particular treatment who show an enhanced response to treatment;
• Non-randomised studies of preventive interventions include disproportionate numbers of individuals who, by virtue of their health related behaviour, have greater capacity to benefit;
• Publication bias leads to negative results being less likely to be published from non-randomised studies than from RCTs.

Exclusions
Randomised controlled trials vary widely in their inclusiveness. Medical reasons cited for exclusion from trials include a high risk of adverse effects and belief that benefit, or lack of it, has already been established for some groups.

Participation
Evaluative research is undertaken predominantly in university or teaching centres, but non-randomised studies are more likely than RCTs to include non-teaching centres, and criteria for participation in RCTs may include the achievement of a specified level of clinical outcome. The available evidence suggests that this may exaggerate the measured treatment effect.⁹

View larger version (20K): [in this window] [in a new window]   Fig 2.    Effect of differences in participation in trials of prevention and of treatment. p=eligible non-participants; shaded areas represent the study population

Impact of patients' preferences
There is little empirical research on the impact on outcome of patients' preferences. The four studies that attempted to measure preference effects either were small or have yet to report full results.^13-16 In theory, preference could have an important impact on results of RCTs, especially where the true effect is small. Such effects could account for some observed differences between results of RCTs and non-randomised studies. There are methods that may detect preference effects reliably; though these may contribute to understanding this phenomenon, none provides a complete answer.¹⁷ This is mainly because randomisation between preferring a treatment and not is impossible, and confounding may bias any observed comparison.

Adjustment for baseline differences in non-randomised studies
Despite the evidence that the results of RCTs and non-randomised studies are often similar, differences in baseline prognostic factors clearly can be important. Absence of randomisation can produce groups that differ in important ways, and it is necessary to consider whether it is possible to adjust for such differences. Adjustment for imbalance in baseline prognostic factors between arms of non-randomised studies commonly changes the size of the measured treatment effect, but such changes are often small and inconsistent.¹

	Recommendations

A large, inclusive, fully blinded RCT incorporating appropriate subgroup analysis is likely to provide the best possible evidence of effectiveness, but there will always be circumstances in which randomisation, especially on an inclusive basis, is unethical or impractical.¹⁸ In circumstances where there are genuine reasons for not randomising,¹⁹ non-randomised studies can provide useful evidence. In such studies, adjustment for baseline imbalances should always be attempted, as rigorously and extensively as possible, and the procedures should be reported explicitly to help readers' evaluations. However, adjustment cannot be relied on to approximate the prognostic balance of randomisation, given unknown or unmeasurable confounding.

Investigators conducting evaluative research (using any design) must improve the quality of reporting. Authors should define the population to whom they expect their results to be applied; what has been done to ensure that the study population is representative of this wider population, and any evidence of how it differs; whether centres that participated differ from those that declined; and the numbers and characteristics of patients eligible to be included who either were not invited to do so or were invited and declined.

The findings of such studies have implications for the way in which evidence is interpreted. When faced with data from any source, whether randomisation has been used or not, it is important first to pursue alternative (non-causal) explanations thoroughly and examine the possible influence of chance, bias, and confounding, perhaps using sensitivity analyses where feasible.

Where only non-randomised data are available, the potential for allocation bias should be considered and any attempts at risk adjustment should be assessed.

Where only randomised trials are found, preference effects should also be considered. To obtain an uncontaminated estimate of the physiological effect of a treatment, RCTs should be blind to everyone involved, but for many interventions this will be impossible. Also, the advantages of narrowing inclusion criteria to ensure high participation in RCTs should be balanced by the potential need for subgroup analysis. It should not be assumed that a summary result applies to all potential patients.

View larger version (60K): [in this window] [in a new window]   Available from the BMJ bookshop (www.bmjbookshop.com)

When both randomised and non-randomised studies have been conducted it is important to ascertain whether estimates of treatment effect are consistent for patients at similar risk across studies. If so, it may be reasonable to accept the results of the less exclusive non-randomised studies. Differences in results cannot be assumed to be solely due to the presence or lack of randomisationdifferences in study populations, characteristics of the intervention, and the effects of patients' preferences may also affect the results.

Whichever design is used, generalisability needs attention. One approach involves examining the relation between reduction in relative risk (as a measure of effect size) against the percentage of events in the control arm (as an indirect measure of inclusiveness)²⁰; this is sometimes referred to as metaregression.²¹ Where sufficient data are available from RCTs, it may be possible to identify separate measures of benefit and harm. If, as has been shown for giving anticoagulants to prevent stroke, the percentage reduction in relative risk remains constant at all levels of severity and the increase in absolute risk of an adverse effect remains constant, the reduction in absolute risk for a given patient can then be estimated.²²

In conclusion, RCTs and non-randomised studies can provide complementary evidencebut it is important that clinicians using this evidence are aware of the strengths and weaknesses of each method.

	Acknowledgments

This article is adapted from Health Services Research Methods: A Guide to Best Practice, edited by Nick Black, John Brazier, Ray Fitzpatrick, and Barnaby Reeves, published by BMJ Books in 1998.

	Footnotes

Series editor: Nick Black

Funding: This work was supported by a grant from the NHS Health Technology Assessment Programme.

Competing interests: None declared.

website extra: Further references are listed on the BMJ's website www.bmj.com

	References

1.	Britton A, McKee M, Black N, McPherson K, Sanderson C, Bain C. Choosing between randomised and non-randomised studies: a systematic review. Health Technol Assess 1998;2(13)
2.	Kunz R, Oxman AD. The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials. BMJ 1998; 317: 1185-1190[Abstract/Free Full Text].
3.	Horwitz RI, Viscoli CM, Clemens JD, Sadock RT. Developing improved observational methods for evaluating therapeutic effectiveness. Am J Med 1990; 89: 630-638[Medline].
4.	Yusuf S, Held P, Teo KK. Selection of patients for randomized controlled trials: implications of wide or narrow eligibility criteria. Stat Med 1990; 9: 73-86[Medline].
5.	Haynes RB, Dantes R. Patient compliance and the conduct and interpretation of therapeutic trials. Controlled Clin Trials 1987; 8: 12-19[Medline].
6.	Gurwitz JH, Nananda F, Avorn J. The exclusion of the elderly and women from clinical trials on acute myocardial infarction. JAMA 1992; 268: 1417-1422[Abstract].
7.	Jones RH, Kesler K, Phillips HR, Mark DB, Smith PK, Nelson CL, et al. Long-term survival benefits of coronary artery bypass grafting and percutaneous transluminal angioplasty in patients with coronary artery disease. J Thorac Cardiovasc Surg 1996; 11: 1013-1025.
8.	Hartz AJ, Kuhn EM, Pryor DB, Krakauer H, Young M, Heudeberg, et al. Mortality after coronary angioplasty and coronary artery bypass surgery (the National Medicare Experience). Am J Cardiol 1992; 70: 179-185[Medline].
9.	Stukenborg GJ. Comparison of carotid endarterectomy outcomes from randomized controlled trials and medicare administrative databases. Arch Neurol 1997; 54: 826-832[Abstract].
10.	Hunninghake DB, Darby CA, Probstfield JL. Recruitment experience in clinical trials: literature summary and annotated bibliography. Controlled Clin Trials 1987; 8: 6S-30S[Medline].
11.	Barofsky I, Sugarbaker PH. Determinants of patients nonparticipation in randomised clinical trials for the treatment of sarcomas. Cancer Clin Trials 1979; 2: 237-249.
12.	Davies G, Pyke S, Kinmouth AL. Effect of non-attenders on the potential of a primary care programme to reduce cardiovascular risk in the population. BMJ 1994; 309: 1553-1556[Abstract/Free Full Text].
13.	McKay JR, Alterman AI, McLellan T, Snider EC, O'Brien CP. Effect of random versus nonrandom assignment in a comparison of inpatient and day hospital rehabilitation for male alcoholics. J Consulting Clin Psychol 1995; 63: 70-78[Medline].
14.	Nicolaides K, de Lourdes Brizot M, Patel F, Snijders R. Comparison of chorionic villus sampling and amniocentesis for fetal karyotyping at 10-13 weeks' gestation. Lancet 1994; 344: 435-439[Medline].
15.	Torgerson DJ, Klaber-Moffett J, Russell IT. Patient preferences in randomised trials: threat or opportunity. J Health Serv Res Policy 1996; 1: 194-197. [Medline]
16.	Fallowfield LJ, Hall A, Maguire GP, Baum M. Psychological outcomes of different treatment policies in women with early breast cancer outside a clinical trial. BMJ 1990; 301: 575-580.
17.	McPherson K, Britton AR, Wennberg JE. Are randomized controlled trials controlled? Patient preferences and unblind trials. J Roy Soc Med 1997; 90: 652-656[Abstract].
18.	Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ 1996; 312: 1215-1218[Free Full Text].
19.	McPherson K. The Cochrane Lecture. The best and the enemy of the good: randomised controlled trials, uncertainty, and assessing the role of patient choice in medical decision making. J Epidemiol Commun Health 1994; 48: 6-15[Abstract].
20.	Davey Smith G, Song F, Sheldon T. Cholesterol lowering and mortality: the importance of considering initial risk. BMJ 1993; 206: 1367-1373.
21.	Holme I. Cholesterol reduction and its impact on coronary artery disease and total mortality. Am J Cardiol 1995; 76: 10-17C.
22.	Glasziou PP, Irwig LM. An evidence based approach to individualising treatment. BMJ 1995; 311: 1356-1359[Free Full Text].