Pragmatic randomised trials using routine electronic health records: putting them to the testBMJ 2012; 344 doi: https://doi.org/10.1136/bmj.e55 (Published 07 February 2012) Cite this as: BMJ 2012;344:e55
All rapid responses
I have read with interest the article by Staa and colleagues (1) on the potential use of electronic health records to conduct pragmatic randomised clinical trials (RCT). To my knowledge, the idea of randomized database studies (RDS) was originally published in 1998 (2), as a possible solution to minimize the generalizability problems of RCT and the confounding problems of observational research.
Some years later, Mosis and colleagues assessed the technical feasibility and the methodological challenges of the idea (3). The authors created software to automatically identify, recruit and randomise patients treated with diclofenac and celecoxib for osteoarthritis. Time required to obtain patients’ inform consent was identified as the major barrier of the study. This obstacle was consistent with the recent findings by Staa and colleagues. Regulatory oversight “even for trials comparing two interventions that have already been shown to be safe and are in widespread and routine use” (1), in situations where clinical equipoise exists, is the main challenge identified by Staa et colleagues.
The combination of experiments and observations in daily practice requires important regulatory and cultural changes oriented to eliminate the barriers between clinical practice and clinical research, realizing that all research and all clinical actions begin at the patient’s bedside (4) and that every medical act is structured like an experiment (5). The increasing use of EHR should contribute to the disappearance of walls between doctors who carry out research, and doctors who do not, between patients who participate in RCTs and the “real” patients that doctors see every day, between the clinical research form used in RCTs and the electronic medical history. The systematic use of EHR to conduct RCTs could contribute to completing the circle that began with evidence-based medicine with the new concept of medicine-based evidence (6). The real challenge is not the technical infrastructure to implement randomized database studies, but understanding that, in the context of a patient-centered medicine, clinical research and clinical practice are the two faces of the same coin.
1. Van Staa T-P, Goldacre B, Gulliford M, Cassell J, Pirmohamed M, et al. Pragmatic randomised trials using routine electronic health records. BMJ 2012;344;e55
2. Sacristan JA, Soto J, Galende I, Hylan TR. Randomized database studies: a new method to assess drugs’ effectiveness? J Clin Epidemiol 1998; 51: 713-5.
3. Mosis G, Dielman JP, Stricker BC, van der Lei J, Sturkenboom MCJM. A randomized database study in general practice yielded quality data but patient recruitment in routine consultation was not practical. J Clin Epidemiol 2006; 59: 497-502.
4. Jenicek M: Clinical case reports: sources of boredom or valuable pieces of evidence? Nat J India 2001; 14:193-4.
5. Feinstein AR: Clinical judgment revisited: the distraction of quantitative models. Ann Intern Med 1994; 120:799-805.
6. Sacristan JA. Exploratory trials, confirmatory observations: a new reasoning model in the era of patient-centered medicine. BMC Med Res Method 2011; 11: 57.
Competing interests: JAS is an employee of Eli Lilly & Company
We fully agree the proposition of Tjeerd-Pieter van Staa and colleagues to promote randomised evaluation of accepted choices in treatment (REACT) using electronic health records1. But we would like to focus on a limitation of their trials design. After randomisation, the various doses, durations, stops, resumes, cross-over and individual modalities of the studied treatments as the different follows-up of the patients may trouble the demonstration of differences between the two compared treatments. The necessary “intention to treat” analysis may often conclude to an absence of difference between the two therapeutic strategies not because of an evidence-based clinical equivalence but because a weak evaluation of the efficacy or safety due to a too large variability linked to the erratic therapeutics compliance in usual conditions. This variability reduces the possibility to demonstrate the superiority of one of the tested treatment even if this superiority exists. Any conclusion about the similarity of the two tested treatment might be excessive. In the opposite, in such trials with a correct sample size calculation and an intention to treat analysis, the evidence of superiority in term of efficacy or safety is probably true, but this situation is rare when the trial compares two commonly used treatments with the same indication.
Jean-François Bergmann professor of therapeutics, Université Paris-Diderot and Hôpital Lariboisière, 75010 Paris, France firstname.lastname@example.org
Celia Lloret-Linares senior lecturer in therapeutics, Université Paris-Diderot and Hôpital Lariboisière, 75010 Paris, France
Amanda Lopes senior lecturer in therapeutics, Université Paris-Diderot and Hôpital Lariboisière, 75010 Paris, France
No competing interest
1 Van Staa TP, Goldacre B, Gulliford M, Cassell J, Pirmohamed M, Taweel A, et al. Pragmatic randomised trials using routine electronic health records. BMJ 2012; 344:e55 (11 February.)
Competing interests: No competing interests
van Staa and colleagues have drawn attention to a most important issue. The potential of data collected at the point of care to support prospective research by randomised controlled trial was highlighted 10 years ago in three monographs published in Health Technology Assessment[1-3]. These demonstrated the feasibility and potential benefits of using routinely collected data to support research but also highlighted the significant problems with data completeness and validity that need to be addressed.
The REACT trials that van Staa and colleagues describe are designed to explore the feasibility of using routinely collected data from primary care and hospitals to answer questions about the efficacy of treatments in primary care. High-level efficacy outcomes such as peak flow rates and cholesterol levels are being measured. The real benefits from the approach will accrue when questions about clinical-effectiveness and cost-effectiveness can also be answered using routinely collected data. This will require either collection of patient-recorded outcome measures (generic measures such as SF-36, disease-specific measures and those from which can be derived quality adjusted life years such as EQ5D) or methods which use clinical data to derive surrogate scores from these scales.
There are in essence five different ways in which research questions about practice in the busy clinical environment could be answered:
1 Conventionally-designed pragmatic trials requiring designed data collection as part of the trial.
2 The use of routine data without randomisation, preferably linking multiple sources. These observational studies can be very large.
3 Randomised trials, using routine data, such as the REACT trials, which are not truly pragmatic but tend to measure efficacy.
4 Hybrid trials, where routine data are supplemented by additional data to measure patient outcomes and cost-effectiveness. This would either require direct recording by patients or development of proxy measurements using routinely collected data.
5 Genuinely pragmatic trials which use routine data for clinicians and patients to measure clinical-effectiveness and cost-effectiveness, without the need for additional designed data collection.
The REACT trials are addressing and will explore many important process issues relating to randomisation and consent but the real breakthrough will come when genuinely pragmatic trials, which use routine data to enable clinicians and patients to measure clinical-effectiveness and cost-effectiveness without the need for additional designed data collection. This goal is presently constrained by both the content and quality of routinely collected clinical data[8,9] but the Academy of Medical Royal Colleges has set out a vision for the future to improve this.
1 Williams JG, Cheung WY, Cohen D, Hutchings H, Longo M, Russell IT. The value of routine data in health technology assessment: can randomised trials rely on existing electronic data? Health Technology Assessment 2003;vol 7:no 26.
2 Lewsey JD, Leyland AH, Murray GD, Boddy FA. Using routine data to complement and enhance the results of randomised controlled trials. Health Technology Assessment 2000;vol 4:no 22.
3 Raftery J, Roderick P, Stevens A. Potential use of routine databases in health technology assessment. Health Technology Assessment 2005;vol 9:no 20.
4 Hutchings HA, Cheung WY, Williams JG, Cohen D, Longo MF, Russell IT. Can electronic routine data act as a surrogate for patient-assessed outcome measures? Int J of Technology Assessment in Health Care 2005;21(1):138-43.
5 Roland M, Torgeson DJ. Understanding controlled trials: what are pragmatic trials? BMJ 1998;316:285.
6 Secure Anonymised Information Linkage (SAIL) databank. http://www.healthinformaticsresearchlabs.swansea.ac.uk/SAILproject.html.
7 van Staa TP, Goldacre B, Gulliford M, Cassell J, Pirmohamed M, Taweel A, Delaney B, Smeeth L. Pragmatic randomised trials using routine electronic health records: putting them to the test. BMJ 2012;344:e55.
8 Williams JG, Mann RY. Hospital Episode Statistics: time for clinicians to get involved? Clinical Medicine 2002;2(1):34-7.
9 Williams JG. Monitoring clinical activity and performance: how can hospital episode statistics be made fit for purpose? Frontline Gastroenterology 2011;2:160-1.
10 The case and vision for patient focused records. Academy of Medical Royal Colleges http://aomrc.org.uk/publications/statements.html).
Competing interests: No competing interests
Dazzled by the “wealth of information” in the electronic health-care record databases, van Staa et al. seek a “revolution” in medical research in the form of a widespread adoption of pragmatic randomised trials.  These studies, it seems, will be more user-friendly, cheaper and have ethical advantages over more conventional research, as well as improving patient care. But we should be wary of following this path.
Pragmatic randomised trials involve a loosening of the procedures put in place to improve the reliability of conclusions drawn from the data of conventional RCTs. This aspect will no doubt come to light when the results of trials using this new method clash with the views of those with a vested interest in particular therapies.
For example, the absence of double-blinding may lead to bias in relation to both the assessment of the end-point and medical decisions made during the study which affect the outcome. Of course, these and other similar problems may be addressed but this would deprive the advocates of pragmatic trials of what they consider to be one of the most attractive features – namely, that the research is conducted under the “usual conditions”. It would appear, therefore, that they must choose between the validity of the research and the supposed advantages of doing the trial in the everyday clinical setting.
The authors make much of their contention that pragmatic trials will allow the data to be generalised. So much so that, in their conclusions, they imply that the external validity of the results will not be open to question. Given that there will be differences between doctors who participate in this research and those who do not, that not all of the eligible patients will be recruited, and that the trial sample will not be drawn randomly from the underlying population, this is very hard to accept.
And there are other problems with the method of research advocated by van Staa et al.  including, for instance, doubts about the cost of the programme and the commitment of doctors not versed in research, especially if pragmatic trials are a success in the eyes of NHS managers and participation becomes a compulsory target. At a more fundamental level, however, these studies will increase the heterogeneity already present in current clinical trials and take us in the direction of an even greater dependence upon statistics.
We have had to endure one revolution in medical research already – the statistical approach involving large-scale RCTs and epidemiological studies – which has generated little of value for patients.  Now we are being asked to endure another with equally little prospect of any genuine benefit. We should learn from our past mistakes.
1. Van Staa T-P, Goldacre B, Gulliford M, Cassell J, Pirmohamed M, et al. Pragmatic randomised trials using routine electronic health records. BMJ 2012;344;e55.
2. Penston J. Stats.con – How we’ve been fooled by statistics-based research in medicine. The London Press. November, 2010.
Competing interests: No competing interests