BMJ  2005;330:765 (2 April), doi:10.1136/bmj.38398.500764.8F (published 14 March 2005)

Information in practice

Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success

¹ Division of Clinical Informatics, Department of Community and Family Medicine, Box 2914, Duke University Medical Center, Durham, NC 27710, USA, ² College of Health Sciences, Old Dominion University, Norfolk, VA 23529, USA

Abstract

Objective To identify features of clinical decision support systems critical for improving clinical practice.

Design Systematic review of randomised controlled trials.

Data sources Literature searches via Medline, CINAHL, and the Cochrane Controlled Trials Register up to 2003; and searches of reference lists of included studies and relevant reviews.

Study selection Studies had to evaluate the ability of decision support systems to improve clinical practice.

Data extraction Studies were assessed for statistically and clinically significant improvement in clinical practice and for the presence of 15 decision support system features whose importance had been repeatedly suggested in the literature.

Results Seventy studies were included. Decision support systems significantly improved clinical practice in 68% of trials. Univariate analyses revealed that, for five of the system features, interventions possessing the feature were significantly more likely to improve clinical practice than interventions lacking the feature. Multiple logistic regression analysis identified four features as independent predictors of improved clinical practice: automatic provision of decision support as part of clinician workflow (P < 0.00001), provision of recommendations rather than just assessments (P = 0.0187), provision of decision support at the time and location of decision making (P = 0.0263), and computer based decision support (P = 0.0294). Of 32 systems possessing all four features, 30 (94%) significantly improved clinical practice. Furthermore, direct experimental justification was found for providing periodic performance feedback, sharing recommendations with patients, and requesting documentation of reasons for not following recommendations.

Conclusions Several features were closely correlated with decision support systems' ability to improve patient care significantly. Clinicians and other stakeholders should implement clinical decision support systems that incorporate these features whenever feasible and appropriate.

Introduction

Recent research has shown that health care delivered in industrialised nations often falls short of optimal evidence based care. To address this, healthcare organisations are increasingly turning to clinical decision support systems, which provide clinicians with patient-specific assessments or recommendations to aid clinical decision making.¹ Such systems have been shown to improve prescribing practices,^2-4 reduce serious medication errors,^{5 6} enhance the delivery of preventive care services,^{7 8} and improve adherence to recommended care standards.^{1 9} Compared with other approaches to improve practice, these systems have also generally been shown to be more effective and more likely to result in lasting improvements.

Clinical decision support systems do not always improve clinical practice. In a recent systematic review of computer based systems, most (66%) significantly improved clinical practice, but 34% did not.¹ Relatively little scientific evidence is available to explain why systems succeed or fail.^{10 11} Some investigators have tried to identify system features most important for improving clinical practice, but they have typically relied on the opinion of a limited number of experts, and none has combined a systematic literature search with quantitative meta-analysis. We therefore systematically reviewed the literature to identify the specific features of clinical decision support systems most crucial for improving clinical practice.

Methods

Data sources
We searched Medline (1966-2003), CINAHL (1982-2003), and the Cochrane Controlled Trials Register (2003). (See bmj.com for search terms.) We also systematically searched the reference lists of included studies and relevant reviews.

Inclusion and exclusion criteria
We defined a clinical decision support system as any electronic or non-electronic system designed to aid directly in clinical decision making, in which characteristics of individual patients are used to generate patient specific assessments or recommendations that are then presented to clinicians for consideration.¹ Our inclusion criteria were any randomised controlled trial evaluating the ability of a clinical decision support system to improve an important clinical practice in a real clinical setting; use of the system by clinicians directly involved in patient care; and assessment of improvements in clinical practice through patient outcomes or process measures. Our exclusion criteria were less than seven units of randomisation per study arm; study not in English; mandatory compliance with decision support system; lack of description of decision support content or of clinician interaction with system; and score of less than five points on a 10 point scale assessing five potential sources of study bias.¹

Study selection
Two authors independently reviewed the titles, index terms, and abstracts of the identified references and rated each paper as "potentially relevant" or "not relevant" using a screening algorithm based on study type, study design, subjects, setting, and intervention. Two authors then independently reviewed the full texts of the selected articles and again rated each paper as "potentially relevant" or "not relevant" using the screening algorithm. Finally, two authors independently applied the full set of inclusion and exclusion criteria to the potentially relevant studies to select the final set of included studies. Disagreements between reviewers were resolved by discussion, and we measured inter-rater agreement.

Data abstraction
A study may include several trial arms, so that multiple comparisons may exist within the single study. For each relevant comparison, two reviewers independently assessed whether the clinical decision support system resulted in an improvement in clinical practice that was both statistically and clinically significant. In some cases changes in practice characterised as clinically significant by the study authors were deemed non-significant by the reviewers.

Next, two reviewers independently determined the presence or absence of specific features of decision support systems that could potentially explain why a system succeeded or failed. We limited our consideration to features that were identified as being potentially important by at least three different sources from the literature, which left us with 22 potential explanatory features, including general system features, system-clinician interaction features, communication content features, and auxiliary features (see bmj.com). Of these, 15 could be included in our analysis because their presence or absence could be reliably abstracted from most studies.

Data synthesis
Univariate analyses—For each of the 15 selected features we individually determined whether interventions possessing the feature were significantly more likely to succeed than interventions lacking the feature. We calculated 95% confidence intervals for individual success rates and for differences in success rates.

Multiple logistic regression analyses—For these analyses, the presence or absence of a statistically and clinically significant improvement in clinical practice constituted the binary outcome variable, and the presence or absence of specific decision support system features constituted binary explanatory variables. We included only cases in which the clinical decision support system was compared against a true control group. (See bmj.com for details.)

Direct experimental evidence—We systematically identified studies in which the effectiveness of a given decision support system was directly compared with the effectiveness of the same system with additional features. We considered a feature to have direct experimental evidence supporting its importance if its addition resulted in a statistically and clinically significant improvement in clinical practice.

Results

Description of studies
Of 10 688 potentially relevant articles screened, 88 papers describing 70 studies met all our inclusion and exclusion criteria. The 70 studies contained 82 relevant comparisons, of which 71 compared a clinical decision support system with a control group (control-system comparisons) and 11 directly compared a system with the same system plus extra features (system-system comparisons). We used the control-system comparisons to identify system features statistically associated with successful outcomes and the system-system comparisons to identify features with direct experimental evidence of their importance.

The most common types of decision support system were computer based systems that provided patient-specific advice on printed encounter forms or on printouts attached to charts (34%), non-electronic systems that attached patient-specific advice to appropriate charts (26%), and systems that provided decision support within computerised physician order entry systems (16%).

Univariate analyses of clinical decision support system features
Overall, 48 of the 71 decision support systems (68% (95% confidence interval 56% to 78%)) significantly improved clinical practice. For five of the 15 features, the success rate of interventions possessing the feature was significantly greater than that of interventions lacking the feature (table 1).

View this table: [in this window] [in a new window]   Table 1 Success rates* of clinical decision support systems (CDSS) with and without 15 potentially important features. Results of 71 control-CDSS comparisons

 

Interventions succeeded when the decision support was provided to clinicians automatically; when systems were provided as an integrated component of charting or order entry systems; when systems used a computer to generate the decision support; when systems prompted clinicians to record a reason when not following the advised course of action; and when systems provided a recommendation. Finally, systems that provided decision support at the point of care were substantially more likely to succeed than systems that did not, but the difference in success rates fell just short of being significant.

Meta-regression analysis
We conducted multivariate logistic regression analyses in order to identify independent predictors of clinical decision support system effectiveness while taking into consideration the presence of other potentially important factors (table 2).

View this table: [in this window] [in a new window]   Table 2 Features of clinical decision support systems (CDSS) associated with improved clinical practice. Results of meta-regression analyses of 71 control-CDSS comparisons

 

Of the six features shown to be important by the univariate analyses, four were identified as independent predictors of system effectiveness by the primary meta-regression analysis. Most notably, this analysis confirmed the critical importance of automatically providing decision support as part of clinician workflow. The other three features were providing decision support at the time and location of decision making, providing a recommendation rather than just an assessment, and using a computer to generate the decision support. Among the 32 clinical decision support systems incorporating all four features, 30 (94% (80% to 99%)) significantly improved clinical practice. In contrast, clinical decision support systems lacking any of the four features improved clinical practice in only 18 out of 39 cases (46% (30% to 62%)).

Survey of direct experimental evidence
We identified 11 randomised controlled trials in which a clinical decision support system was evaluated directly against the same clinical decision support system with additional features. In support of the regression results, one study found that system effectiveness was significantly enhanced when the decision support was provided at the time and location of decision making. Similarly, effectiveness was enhanced when clinicians were required to document the reason for not following system recommendations and when clinicians were provided with periodic feedback about their compliance with system recommendations. Furthermore, two of four studies found a significant beneficial effect when decision support results were provided to both clinicians and patients. In contrast, clinical decision support system effectiveness remained largely unchanged when critiques were worded more strongly and the evidence supporting the critiques was expanded to include institution-specific data, when recommendations were made more specific, when local clinicians were recruited into the system development process, and when bibliographic citations were provided to support the recommendations made by the system.

Discussion

We identified 22 technical and non-technical factors repeatedly suggested in the literature as important determinants of a clinical support system's ability to improve clinical practice, and we evaluated 15 of these features in randomised controlled trials of clinical decision support systems. We found five of the features were significantly correlated with system success, and one feature correlated with system success at just over the 0.05 significance level. Multiple logistic regression analysis identified four of these features as independ-ent predictors of a system's ability to improve clinical practice. Furthermore, we found direct experimental evidence to support the importance of three additional features.

What is already known on this topic Clinical decision support systems have shown great promise for reducing medical errors and improving patient care Such systems do not always result in improved clinical practice, for reasons that are not always clear What this study adds Four features are strongly associated with a decision support system's ability to improve clinical practice—(a) decision support provided automatically as part of clinician workflow, (b) decision support delivered at the time and location of decision making, (c) actionable recommendations provided, and (d) computer based All four features make it easier for clinicians to use a clinical decision support system, suggesting that an effective system must minimise the effort required by clinicians to receive and act on system recommendations

Strengths and limitations of study
Strengths include our thorough literature search and systematic review for relevant expert opinion, the use of two independent reviewers for study selection and data abstraction to increase the reliability of our findings, provision of a quantitative estimate of the relative importance of specific clinical decision support system features and, finally, provision of a comprehensive summary of randomised controlled trials.

One limitation of this study is that we used a binary outcome measure rather than a continuous measure such as effect size. We therefore could not adjust for variations in the size of outcomes. Another potential criticism is that we pooled different types of clinical decision support systems in the regression analysis. However, we believe that our methods were appropriate given that our objective was to determine the impact of heterogeneity among interventions rather than to estimate the effects of a homogeneous intervention, as is usually the case for a meta-analysis.

Our analyses were limited to published reports of randomised controlled trials. Thus they may not be extendable to categories where we could not find any studies, such as those provided on personal digital assistants. Also, publication bias against studies that failed to show an effect might have limited our ability to identify features associated with ineffective systems.

The sample size for our regression analysis was restricted by the size of the available literature. Thus, our ratio of cases to explanatory variables was suboptimal, especially for the subset regression analyses. As a result, we cannot rule out the importance of system features based on their absence from the final regression models. Also, it is possible that one or more features were falsely included into the regression models because of over-fitting. However, our findings are consistent with our previous experiences of implementing clinical decision support systems in practice. An additional limitation is that our analyses were restricted to features that could be reliably abstracted.

Implications
On a practical level, our findings imply that clinicians should implement clinical decision support systems that (a) provide decision support automatically as part of clinician workflow, (b) deliver decision support at the time and location of decision making, (c) provide actionable recommendations, and (d) use a computer to generate the decision support. As a general principle our findings suggest that an effective clinical decision support system must minimise the effort required by clinicians to receive and act on system recommendations. When feasible and appropriate, clinical decision support systems should also provide periodic performance feedback, request documentation of the reason for not following system recommendations, and share decision support results with patients.

Future directions
The promise of evidence based medicine will be fulfilled only when strategies for implementing best practice are rigorously evidence based themselves. Two important research needs must still be addressed. Firstly, reports of clinical decision support system evaluations should provide as much detail as possible when describing the systems and the manner in which clinicians interacted with them, so that others can learn more effectively. Secondly, further direct experimentation is warranted to evaluate the importance of specific system features.

---

References w1-w88, the studies reviewed in this article, are on bmj.com

This is the abridged version of an article that was posted on bmj.com on 14 March 2005: http://bmj.com/cgi/doi/10.1136/bmj.38398.500764.8F

Editorial by Purcell

We thank Vic Hasselblad for his assistance with the statistical analyses.

Contributors and guarantor: See bmj.com

Funding: This study was supported by research grants T32-GM07171 and F37-LM008161-01 from the National Institutes of Health, Bethesda, Maryland, USA; and by research grants R01-HS10472 and R03-HS10814 from the Agency for Healthcare Research and Quality, Rockville, Maryland, USA. These funders did not play a role in the design, execution, analysis, or publication of this study.

Competing interests: None declared.

Ethical approval: Not required.

References

1. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998;280: 1339-46.[Abstract/Free Full Text]
2. Bennett JW, Glasziou PP. Computerised reminders and feedback in medication management: a systematic review of randomised controlled trials. Med J Aust 2003;178: 217-22.[ISI][Medline]
3. Walton RT, Harvey E, Dovey S, Freemantle N. Computerised advice on drug dosage to improve prescribing practice. Cochrane Database Syst Rev 2001;1: CD002894.
4. Walton R, Dovey S, Harvey E, Freemantle N. Computer support for determining drug dose: systematic review and meta-analysis. BMJ 1999;318: 984-90.[Abstract/Free Full Text]
5. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003;163: 1409-16.[Abstract/Free Full Text]
6. Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma'Luf N, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999;6: 313-21.[Abstract/Free Full Text]
7. Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc 1996;3: 399-409.[Abstract/Free Full Text]
8. Balas EA, Weingarten S, Barb CT, Blumenthal D, Boren SA, Brown GD. Improving preventive care by prompting physicians. Arch Intern Med 2000;160: 301-8.[Abstract/Free Full Text]
9. Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc 1999;6: 104-14.[Abstract/Free Full Text]
10. Kaplan B. Evaluating informatics applications—some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inf 2001;64: 39-56.[CrossRef][ISI][Medline]
11. Kanouse DE, Kallich JD, Kahan JP. Dissemination of effectiveness and outcomes research. Health Policy 1995;34: 167-92.[CrossRef][ISI][Medline]

(Accepted 14 February 2005)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Relevant Article

What makes a good clinical decision support system

Gretchen P Purcell
BMJ 2005 330: 740-741. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

• Hemingway, H., Chen, R., Junghans, C., Timmis, A., Eldridge, S., Black, N., Shekelle, P., Feder, G. (2008). Appropriateness Criteria for Coronary Angiography in Angina: Reliability and Validity. ANN INTERN MED 149: 221-231 [Abstract] [Full text]  
• Kilbridge, P. M., Classen, D. C. (2008). The Informatics Opportunities at the Intersection of Patient Safety and Clinical Informatics. J. Am. Med. Inform. Assoc. 15: 397-407 [Abstract] [Full text]  
• Varonen, H., Kortteisto, T., Kaila, M., for the EBMeDS Study Group, (2008). What may help or hinder the implementation of computerized decision support systems (CDSSs): a focus group study with physicians. Fam Pract 25: 162-167 [Abstract] [Full text]  
• van Wyk, J. T., van Wijk, M. A.M., Sturkenboom, M. C.J.M., Mosseveld, M., Moorman, P. W., van der Lei, J. (2008). Electronic Alerts Versus On-Demand Decision Support to Improve Dyslipidemia Treatment: A Cluster Randomized Controlled Trial. Circulation 117: 371-378 [Abstract] [Full text]  
• Roukema, J., Steyerberg, E. W., van der Lei, J., Moll, H. A. (2008). Randomized Trial of a Clinical Decision Support System: Impact on the Management of Children with Fever without Apparent Source. J. Am. Med. Inform. Assoc. 15: 107-113 [Abstract] [Full text]  
• Fiks, A. G., Grundmeier, R. W., Biggs, L. M., Localio, A. R., Alessandrini, E. A. (2007). Impact of Clinical Alerts Within an Electronic Health Record on Routine Childhood Immunization in an Urban Pediatric Population. Pediatrics 120: 707-714 [Abstract] [Full text]  
• Harrison, M. I., Koppel, R., Bar-Lev, S. (2007). Unintended Consequences of Information Technologies in Health Care An Interactive Sociotechnical Analysis. J. Am. Med. Inform. Assoc. 14: 542-549 [Abstract] [Full text]  
• Vermeulen, L. C., Rough, S. S., Thielke, T. S., Shane, R. R., Ivey, M. F., Woodward, B. W., Pierpaoli, P. G., Thomley, S. M., Borr, C. A., Zilz, D. A. (2007). Strategic approach for improving the medication-use process in health systems: The high-performance pharmacy practice framework. Am J Health Syst Pharm 64: 1699-1710 [Abstract] [Full text]  
• Charney, P. (2007). Computer Technology and the Nutrition Support Professional: Make It Work For You!. Nutr Clin Pract 22: 421-427 [Abstract] [Full text]  
• Klebe, B., Farmer, C., Cooley, R., de Lusignan, S., Middleton, R., O'Donoghue, D., New, J., Stevens, P. (2007). Kidney disease management in UK primary care: guidelines, incentives and Information Technology. Fam Pract 24: 330-335 [Abstract] [Full text]  
• Wright, A., Goldberg, H., Hongsermeier, T., Middleton, B. (2007). A Description and Functional Taxonomy of Rule-based Decision Support Content at a Large Integrated Delivery Network. J. Am. Med. Inform. Assoc. 14: 489-496 [Abstract] [Full text]  
• Crosson, J. C., Ohman-Strickland, P. A., Hahn, K. A., DiCicco-Bloom, B., Shaw, E., Orzano, A. J., Crabtree, B. F. (2007). Electronic Medical Records and Diabetes Quality of Care: Results From a Sample of Family Medicine Practices. Ann Fam Med 5: 209-215 [Abstract] [Full text]  
• Voorham, J., Denig, P., Groningen Initiative to Analyse Type 2 Diabetes Tr, (2007). Computerized Extraction of Information on the Quality of Diabetes Care from Free Text in Electronic Patient Records of General Practitioners. J. Am. Med. Inform. Assoc. 14: 349-354 [Abstract] [Full text]  
• Boord, J. B., Sharifi, M., Greevy, R. A., Griffin, M. R., Lee, V. K., Webb, T. A., May, M. E., Waitman, L. R., May, A. K., Miller, R. A. (2007). Computer-based Insulin Infusion Protocol Improves Glycemia Control over Manual Protocol. J. Am. Med. Inform. Assoc. 14: 278-287 [Abstract] [Full text]  
• Chauhan, U. (2007). Cardiovascular disease prevention in primary care. Br Med Bull 0: ldm002v1-15 [Abstract] [Full text]  
• Dorr, D., Bonner, L. M., Cohen, A. N., Shoai, R. S., Perrin, R., Chaney, E., Young, A. S. (2007). Informatics Systems to Promote Improved Care for Chronic Illness: A Literature Review. J. Am. Med. Inform. Assoc. 14: 156-163 [Abstract] [Full text]  
• Kawamoto, K., Lobach, D. F. (2007). Proposal for Fulfilling Strategic Objectives of the U.S. Roadmap for National Action on Decision Support through a Service-oriented Architecture Leveraging HL7 Services. J. Am. Med. Inform. Assoc. 14: 146-155 [Abstract] [Full text]  
• Osheroff, J. A., Teich, J. M., Middleton, B., Steen, E. B., Wright, A., Detmer, D. E. (2007). A Roadmap for National Action on Clinical Decision Support. J. Am. Med. Inform. Assoc. 14: 141-145 [Abstract] [Full text]  
• Rubin, M. A., Bateman, K., Donnelly, S., Stoddard, G. J., Stevenson, K., Gardner, R. M., Samore, M. H. (2006). Use of a Personal Digital Assistant for Managing Antibiotic Prescribing for Outpatient Respiratory Tract Infections in Rural Communities. J. Am. Med. Inform. Assoc. 13: 627-634 [Abstract] [Full text]  
• Gandhi, T. K., Kachalia, A., Thomas, E. J., Puopolo, A. L., Yoon, C., Brennan, T. A., Studdert, D. M. (2006). Missed and delayed diagnoses in the ambulatory setting: a study of closed malpractice claims.. ANN INTERN MED 145: 488-496 [Abstract] [Full text]  
• Wachter, R. M. (2006). Is ambulatory patient safety just like hospital safety, only without the "stat"?. ANN INTERN MED 145: 547-549 [Full text]  
• Grindrod, K. A, Patel, P., Martin, J. E (2006). What Interventions Should Pharmacists Employ to Impact Health Practitioners' Prescribing Practices?. The Annals of Pharmacotherapy 40: 1546-1557 [Abstract] [Full text]  
• Glazier, R. H., Bajcar, J., Kennie, N. R., Willson, K. (2006). A Systematic Review of Interventions to Improve Diabetes Care in Socially Disadvantaged Populations. Diabetes Care 29: 1675-1688 [Abstract] [Full text]  
• Drew, R. H., Kawamoto, K., Adams, M. B. (2006). Information technology for optimizing the management of infectious diseases. Am J Health Syst Pharm 63: 957-965 [Full text]  
• Morrison, R. S., Meier, D. E., Fischberg, D., Moore, C., Degenholtz, H., Litke, A., Maroney-Galin, C., Siu, A. L. (2006). Improving the management of pain in hospitalized adults.. Arch Intern Med 166: 1033-1039 [Abstract] [Full text]  
• Berner, E. S., Houston, T. K., Ray, M. N., Allison, J. J., Heudebert, G. R., Chatham, W. W., Kennedy, J. I. Jr., Glandon, G. L., Norton, P. A., Crawford, M. A., Maisiak, R. S. (2006). Improving Ambulatory Prescribing Safety with a Handheld Decision Support System: A Randomized Controlled Trial. J. Am. Med. Inform. Assoc. 13: 171-179 [Abstract] [Full text]  
• Reilly, B. M., Evans, A. T. (2006). Translating Clinical Research into Clinical Practice: Impact of Using Prediction Rules To Make Decisions. ANN INTERN MED 144: 201-209 [Abstract] [Full text]  
• Jimbo, M., Nease, D. E. Jr, Ruffin, M. T. IV, Rana, G. K. (2006). Information Technology and Cancer Prevention. CA Cancer J Clin 56: 26-36 [Abstract] [Full text]  
• Samore, M. H., Bateman, K., Alder, S. C., Hannah, E., Donnelly, S., Stoddard, G. J., Haddadin, B., Rubin, M. A., Williamson, J., Stults, B., Rupper, R., Stevenson, K. (2005). Clinical Decision Support and Appropriateness of Antimicrobial Prescribing: A Randomized Trial. JAMA 294: 2305-2314 [Abstract] [Full text]  
• Purcell, G. P (2005). What makes a good clinical decision support system. BMJ 330: 740-741 [Full text]  

Rapid Responses:

Read all Rapid Responses

Decision support systems are beneficial for interventions on specific populations.

Elizabeth A Garthwaite, et al.
bmj.com, 7 Apr 2005 [Full text]