Education and debate

Using high quality clinical databases to complement the results of randomised controlled trials: the case of recombinant human activated protein C

Andrew Padkin, MRC training fellow in health services research ^a, Kathy Rowan, director ^b, Nick Black, professor of health services research ^a. 

^a Editorial by Hinds Health Services Research Unit, Department of Public Health and Policy, London School of Hygiene and Tropical Medicine, London WC1E 7HT, ^b Intensive Care National Audit and Research Centre, Tavistock House, London WC1H 9HR

Correspondence to: A Padkin Andrew.Padkin{at}lshtm.ac.uk

Understanding the generalisability (or applicability) of the results of randomised controlled trials in typical clinical practice remains one of the key methodological challenges to achieving a more scientific basis for health care.¹ Little effort is made to use a scientific approach to assess generalisability, document the use of a new intervention systematically, or determine whether the trials' results are replicated in real life, either in the original patient groups studied or in other patients who receive the intervention.

To explore these issues, we use a case study to describe the practical difficulties that exist for policymakers and clinicians in interpreting the results of a randomised controlled trial evaluating recombinant human activated protein C, a new drug for treating severe sepsis in intensive care patients. We suggest how an existing, high quality, clinical database could provide information on the generalisability of the trial results and on the likely financial consequences of the drug's introduction, and how it could be used to monitor the diffusion and effectiveness of the drug in typical clinical practice.

	Limitations of randomised controlled trials

Evidence based medicine has focused on understanding the factors affecting the internal validity of randomised controlled trials but has paid far less attention to their generalisability. This is reflected in the many instruments for assessing the quality of trials, which concentrate predominantly on identifying factors that may challenge internal validity.² In many randomised controlled trials both the setting and the patients studied differ from those in typical clinical practice. So, even if the evaluated intervention provides significant benefit in the patients studied, whether and for whom it should be used in routine practice remains a matter of judgment. Information is rarely presented or available to help determine an appropriate policy.³ Consequently, the diffusion of a new intervention into routine practice is often haphazard, both within and outside the patient group in which the trial was performed. Even in the highly regulated arena of commercial pharmaceutical research there are many examples of drugs in common use outside their licensed indications. ^{4 5}

	New treatment for severe sepsis

Severe sepsis is common in intensive care and has a high associated mortality. Previous epidemiological studies have been hampered by the multiplicity of definitions used for severe sepsis and the diversity of patient groups studied. The search for a treatment for severe sepsis has been a high priority for many years, but, despite over 40 trials of inflammatory modulators, no effective intervention has been identified.

Recently, results from a major international randomised controlled trial (the PROWESS trial) of a potential agent for treating severe sepsis in intensive care were reported.⁶ The definitions used for severe sepsis (see box) were based on those developed by the American College of Chest Physicians and Society of Critical Care Medicine⁷ and covered a wide spectrum of illness severity. The trial was stopped after interim analysis showed that treatment with recombinant human activated protein C (also called drotrecogin alfa (activated)) was associated with lower 28 day mortality from all causes than placebo (24.7% v 30.8%). This translated into a number needed to treat of 16 to prevent one death by 28 days, considerably lower than the number needed to treat of 56 to prevent one death by 35 days for intravenous thrombolysis in acute myocardial infarction⁸a widely accepted benchmark of effective clinical practice.

As activated protein C heads towards the UK market, the National Institute for Clinical Excellence (NICE) is likely to be asked to decide whether the NHS should provide it and, if so, for which patients. If a decision has not been reached by the time the drug is marketed in early 2002 individual doctors will have to decide on the use of the drug for individual patients.

	Possible policy responses

There are four possible policy responses to the PROWESS trial results: the results could be (a) rejected because they are not deemed valid, (b) rejected because they are not generalisable, (c) accepted as indicating use of the drug for all patients similar to those in the trial, or (d) accepted with more stringent indications as to who should receive the drug. We consider each of these options with respect to defining the further information necessary to make an informed decision and to the way in which a high quality clinical database could help with the decision.

Reject results as invalid
Although, as yet, there have been few challenges to the trial's internal validity, the question as to whether the 28 day survival advantage translates into improved long term survival without a substantive reduction in quality of life remains unanswered. Further follow up of the trial patients by the PROWESS investigators could provide an answer to this question. Without this information, the cost effectiveness of the drug remains uncertain, although it should be remembered that patients with severe sepsis are often young and that recovery from the acute episode is likely to be followed by many years of normal life.

Reject results as not generalisable
Can the trial results be generalised to intensive care practice in Britain? One of the trial's strengths is that it was pragmatic⁹: it did not standardise other interventions and so was designed to investigate the benefit of activated protein C in typical clinical practice. However, despite being conducted in 164 centres in 11 countries, the trial did not include any British centres, the reasons for which are unknown. The generalisability of the results depends on whether the patients studied in the trial can be considered representative of patients with severe sepsis in British intensive care units. This question has two components: whether patients with severe sepsis in British intensive care units are similar to those recruited into the trial, and whether aspects of intensive care other than the use of activated protein C are similar in Britain to those in the trial centres.

View larger version (223K): [in this window] [in a new window]   Severe sepsis covers a wide range of illness severity, from single organ dysfunction (top) to multiple organ dysfunction (bottom). (Reproduced with subjects' permission)

Accept results and recommend drug for all similar patients
The third possible policy response is acceptance of the validity and generalisability of the PROWESS trial results with the recommendation that the drug be used for all patient groups studied in the trial. The financial implications of such a policy are considerable. The absolute number of patients who would be eligible for activated protein C is unknown, but if only half of the 28% of patients admitted to intensive care who met the definition for severe sepsis used in the PROWESS trial were eligible to receive the drug when the trial exclusion criteria were also considered, then 10 000 patients a year might be eligible in England and Wales alone. The manufacturers have not yet revealed the cost of the drug, but HA-1A (Centoxin), another drug marketed for treating severe sepsis in 1992, cost £2200 per treatment course.¹² It is not unreasonable to suppose that activated protein C might cost between £3000 and £5000 per course, leading to a total potential cost of £30m-£50m a year in England and Wales, without including other patients who might also be eligible for treatment.

Accept results but set more stringent limits as to who should receive the drug
It might be suggested that use of the drug should be limited to those patients who would be expected to gain maximum benefit. Unfortunately, this fourth possible policy response would require data that are not available. The only source of such information would be subgroup analyses of the PROWESS trial. The dangers of such analyses are well recognised, and any results would have to be interpreted with great caution.¹³

	Using clinical databases to record typical clinical practice

In situations where a randomised controlled trial is not possible, a non-randomised approach should be considered.¹⁴ A well designed non-randomised study performed to coincide with the launch of activated protein C could not only monitor use of the drug but also provide information on how effective it is in typical clinical practice in predefined subgroups. This would not only supplement the subgroup analysis of data from the PROWESS trial but also permit investigation of those groups not included in the trial but for whom activated protein C will undoubtedly be used (such as patients under 18 years of age). Such a study would need to be large and to be started soon. This could be achieved by the intensive care units that contribute to the case mix programme because the data collection systems already exist and participating clinicians are familiar with the national aggregated database being used for research.¹⁵

	Conclusions

Although this case study describes one particular drug, it illustrates a common problem in healthcare policymaking. Insufficient attention has been paid to the generalisability of the results of randomised controlled trials which, all too often, are conducted with patients and services that are atypical. Fortunately, a high quality clinical database exists for intensive care in England, Wales, and Northern Ireland. This can be used to test the generalisability of the information available on the effectiveness of activated protein C before an evidence based decision is made about its use. The case mix programme database could not only be used to aid the interpretation of PROWESS trial results but also to provide detailed evidence on the likely financial implications in typical clinical practice, to monitor the drug's diffusion into clinical practice, and to measure its clinical effectiveness in the real world. Integrating the results of a randomised controlled trial with those available from a high quality clinical database provides a powerful model for the assessment of healthcare interventions as they pass from trials into clinical practice.

	Acknowledgments

We thank Dr J Nolan of the Royal United Hospital, Bath, for providing the clinical photographs. In addition to his work at the Health Services Research Unit, AP is a consultant in anaesthesia and intensive care medicine at the Royal United Hospital, Bath.

Contributors: This article arose out of discussions between all authors, and all participated in drafting and editing it. AP will act as guarantor.

	Footnotes

Funding: AP is funded by the Medical Research Council. KR is funded by the Intensive Care National Audit and Research Centre. NB is funded by the London School of Hygiene and Tropical Medicine.

Competing interests: AP and KR have received travel and subsistence costs from Eli Lilly to attend conferences on critical care. The Intensive Care National Audit and Research Centre has undertaken analyses on severe sepsis as paid consultancy work for Eli Lilly, and all authors are involved with the centre: KR as director, NB as scientific advisor, and AP as clinical advisor.

	References

1.	Britton A, McKee M, Black N, McPherson K, Sanderson C, Bain C. Threats to applicability of randomised trials: exclusions and selective participation. J Health Serv Res Policy 1999; 4(2): 112-121[Medline].
2.	Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials 1995; 16: 62-73[Medline].
3.	Black N. Evidence based policy: proceed with care. BMJ 2001; 323: 275-279[Free Full Text].
4.	Turner S, Longworth A, Nunn AJ, Choonara I. Unlicensed and off label drug use in paediatric wards: prospective study. BMJ 1998; 316: 343-345[Abstract/Free Full Text].
5.	Mather CM, O'Kelly SW. Unlicensed drug administration. Anaesthesia 1995; 50: 189-190[Medline].
6.	Bernard GR, Vincent J-L, Laterre P-F, LaRosa SP, Dhainaut J-F, Lopez-Rodriguez A, et al. Efficacy and safety of recombinant human activated protein C for severe sepsis. N Engl J Med 2001; 344: 699-709[Abstract/Free Full Text].
7.	Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. Chest 1992; 101: 1644-1655[Abstract/Free Full Text].
8.	Fibrinolytic Therapy Trialists' (FTT) Collaborative Group. Indications for fibrinolytic therapy in suspected acute myocardial infarction: collaborative overview of early mortality and major morbidity results from all randomised trials of more than 1000 patients. Lancet 1994; 343: 311-322[CrossRef][Medline].
9.	Roland M, Torgerson DJ. What are pragmatic trials? BMJ 1998; 316: 285[Free Full Text].
10.	Rowan K, Black N. A bottom-up approach to performance indicators through clinician networks. Health Care UK 2000; Spring: 42-46.
11.	Rowan K. ICNARC sepsis data. In: Programme of the Intensive Care Society Spring meeting. London: Intensive Care Society, 2001:51.
12.	British Medical Association, Royal Pharmaceutical Society of Great Britain. British national formulary. London: BMA, RPS, 1992.(No. 33.)
13.	Assmann SF, Pocock SJ, Enos LE, Kasten LE. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet 2000; 355: 1064-1069[CrossRef][Medline].
14.	Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ 1996; 312: 1215-1218[Free Full Text].
15.	Goldfrad C, Rowan K. Consequences of discharges from intensive care at night. Lancet 2000; 355: 1138-1142[CrossRef][Medline].

(Accepted 7 September 2001)