- Pavel S Roshanov, medical student1,
- Natasha Fernandes, medical student2,
- Jeff M Wilczynski, undergraduate student3,
- Brian J Hemens, doctoral candidate4,
- John J You, assistant professor467,
- Steven M Handler, assistant professor 5,
- Robby Nieuwlaat, assistant professor45,
- Nathan M Souza, doctoral candidate4,
- Joseph Beyene, associate professor45,
- Harriette G C Van Spall, assistant professor67,
- Amit X Garg, professor489,
- R Brian Haynes, professor4710
- 1Schulich School of Medicine and Dentistry, University of Western Ontario, 1151 Richmond St, London, ON, Canada N6A 3K7
- 2Faculty of Medicine, University of Ottawa, 451 Smyth Rd, Ottawa, ON, Canada K1H 8M5
- 3Department of Health, Aging, and Society, McMaster University, 1280 Main St W, Hamilton, ON, Canada L8S 4K1
- 4Department of Clinical Epidemiology and Biostatistics, McMaster University, 1280 Main St W, Hamilton, ON, Canada L8S 4K1
- 5Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, USA
- 6Population Health Research Institute, 237 Barton St E, Hamilton, Canada L8L 2X2
- 7Department of Medicine, McMaster University, 1280 Main St W, Hamilton, ON, Canada L8S 4K1
- 8Department of Medicine, University of Western Ontario, 1151 Richmond St, London, ON, Canada N6A 3K7
- 9Department of Epidemiology and Biostatistics, University of Western Ontario, 1151 Richmond St, London, ON, Canada N6A 3K7
- 10Health Information Research Unit, McMaster University, 1280 Main St W, Hamilton, ON, Canada L8S 4K1
- Correspondence to: R B Haynes, McMaster University, Department of Clinical Epidemiology and Biostatistics, 1280 Main Street West, CRL-133, Hamilton, Ontario, Canada L8S 4K1
- Accepted 17 January 2013
Objectives To identify factors that differentiate between effective and ineffective computerised clinical decision support systems in terms of improvements in the process of care or in patient outcomes.
Design Meta-regression analysis of randomised controlled trials.
Data sources A database of features and effects of these support systems derived from 162 randomised controlled trials identified in a recent systematic review. Trialists were contacted to confirm the accuracy of data and to help prioritise features for testing.
Main outcome measures “Effective” systems were defined as those systems that improved primary (or 50% of secondary) reported outcomes of process of care or patient health. Simple and multiple logistic regression models were used to test characteristics for association with system effectiveness with several sensitivity analyses.
Results Systems that presented advice in electronic charting or order entry system interfaces were less likely to be effective (odds ratio 0.37, 95% confidence interval 0.17 to 0.80). Systems more likely to succeed provided advice for patients in addition to practitioners (2.77, 1.07 to 7.17), required practitioners to supply a reason for over-riding advice (11.23, 1.98 to 63.72), or were evaluated by their developers (4.35, 1.66 to 11.44). These findings were robust across different statistical methods, in internal validation, and after adjustment for other potentially important factors.
Conclusions We identified several factors that could partially explain why some systems succeed and others fail. Presenting decision support within electronic charting or order entry systems are associated with failure compared with other ways of delivering advice. Odds of success were greater for systems that required practitioners to provide reasons when over-riding advice than for systems that did not. Odds of success were also better for systems that provided advice concurrently to patients and practitioners. Finally, most systems were evaluated by their own developers and such evaluations were more likely to show benefit than those conducted by a third party.
We thank Anne M Holbrook for her comments on earlier versions of this manuscripts and Nicholas Hobson for his programming support throughout the project.
Contributors: RBH supervised the study and is guarantor. PSR organised all aspects of the study. PSR, NF, JMW, JJY, NMS, RN, BJH, SMH, HGCVS, JB, and RBH were all involved in the design of the study. PSR drafted the analysis plan and all other authors contributed. PSR, NF, and JMW collected and organised the data. PSR analysed the data. PSR, NF, JMW, JJY, BJH, SMH, HGCVS, AXG, JB, and RBH interpreted the results. PSR and NF wrote the first draft of this manuscript and made subsequent revisions based on comments from JMW, NMS, JJY, BJH, SMH, HGCVS, RN, JB, AXG, and RBH, who reviewed the article for important intellectual content. All authors approved the final manuscript.
Funding: This study was funded by a Canadian Institutes of Health Research Synthesis Grant: Knowledge Translation KRS 91791.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare that PSR and NF were supported by McMaster University, Ontario graduate scholarships, and Canadian Institutes of Health Research “Banting and Best” Canadian graduate scholarships, JJY is supported by a Hamilton Health Sciences Research early career award, and that PSR is a co-applicant for a patent concerning computerised decision support for anticoagulation, which was not involved in this analysis.
Ethical approval: Not required.
Data sharing: Statistical code and dataset are available from the corresponding author.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.