BMJ  2004;329:1017 (30 October), doi:10.1136/bmj.329.7473.1017

Learning in practice

What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review

¹ Education Resource Centre, Birmingham Women's Hospital, Birmingham, B15 2TG

Abstract

Objective To evaluate the effects of standalone versus clinically integrated teaching in evidence based medicine on various outcomes in postgraduates.

Design Systematic review of randomised and non-randomised controlled trials and before and after comparison studies.

Data sources Medline, Embase, ERIC, Cochrane Library, DARE, HTA database, Best Evidence, BEME, and SCI.

Study selection 23 studies: four randomised trials, seven non-randomised controlled studies, and 12 before and after comparison studies. 18 studies (including two randomised trials) evaluated a standalone teaching method, and five studies (including two randomised trials) evaluated a clinically integrated teaching method.

Main outcome measures Knowledge, critical appraisal skills, attitudes, and behaviour.

Results Standalone teaching improved knowledge but not skills, attitudes, or behaviour. Clinically integrated teaching improved knowledge, skills, attitudes, and behaviour.

Conclusion Teaching of evidence based medicine should be moved from classrooms to clinical practice to achieve improvements in substantial outcomes.

Introduction

The knowledge and skills needed for critical appraisal of literature and practice of evidence based medicine (EBM) are often taught through standalone courses and workshops in classrooms away from clinical practice. An early (and now out of date) review showed that in these educational interventions, gains in knowledge were poorer among postgraduates than undergraduates.¹ Without reinforcement in subsequent practice, even the modest knowledge gains from such courses are likely to deteriorate over time. Postgraduate and continuing education received in this way is unlikely to lead to any meaningful changes in clinical care. In theory, teaching and learning that is integrated into routine practice should bring greater benefits.

We examined the effects of postgraduate teaching in EBM and explored the effect of the teaching methods (whether standalone or integrated into clinical practice) on various outcomes.

Methods

We searched Medline, Embase, ERIC, Cochrane controlled trials Register (CCTR), Cochrane database of systematic reviews (CDSR), database of abstracts of reviews of effects (DARE), Health Technology Assessment database (HTA), Best Evidence, Best Evidence Medical Education (BEME), and Science Citation Index (SCI) using the following search terms and their word variants: "evidence", "critical", "appraisal" or "journal club" combined with "AND" to "teach$", "learn$", "instruct$", or "education". We also searched reference lists of known systematic reviews.^1-4 The final electronic search was conducted in April 2004.

We included studies that evaluated the effects of postgraduate EBM or critical appraisal teaching compared with a control group or baseline before teaching, using a measure of participants' learning achievements or patients' health gains as outcomes. Learning achievement was assessed separately for knowledge, critical appraisal skills, attitudes, and behaviour.

We excluded studies on teaching of EBM in undergraduate education. We graded the quality of the evidence in these articles as either level 1 (randomised comparison) or level 2 (non-randomised studies). We could not use meta-analysis because of the obvious heterogeneity in features, quality, and assessment tools in individual studies. We weighted our conclusions by quality of methods.

Results

The literature search identified 42 potentially useful citations. We examined the full manuscripts of all of these citations and identified 23 articles relevant for inclusion in our review. Of the 19 articles that we rejected, 15 examined populations unsuitable for our review (for example, undergraduates or non-medical staff), two examined an unsuitable intervention (for example, the effect of dissemination of EBM guidelines rather than teaching of EBM), and two were reviews of primary studies already included in our review. For details of the included studies please see bmj.com. Of the 23 included studies, four were level 1 and 19 were level 2, comprising seven non-randomised controlled studies and 12 before and after comparison studies. Eighteen studies, including two level 1 studies, evaluated a standalone teaching method. Five studies, including two randomised trials, evaluated an integrated teaching method in which training of EBM was in real time clinical ward rounds, or in sessions based on real and current patient encounters in wards and clinics.

Does knowledge improve?
Of the 23 studies, 17 assessed knowledge (fig 1). The weight of evidence, including the evidence from the three randomised trials that reported on this outcome, indicated an improvement in knowledge from both teaching methods.

View larger version (41K): [in this window] [in a new window]   Fig 1 Changes in knowledge, skills, attitude, and behaviour after critical appraisal skills or EBM teaching, grouped by quality of studies. Data presented as 100% stacked bar chart with numbers inside bars indicating number of studies that provided information for a particular outcome (see bmj.com for details of each study)

 

Do critical appraisal skills improve?
Nine of the studies assessed critical appraisal skills (fig 1). The only randomised trial that reported this outcome in the standalone group did not find an improvement. Of the six non-randomised studies that reported this outcome in the standalone group, three found an improvement. On the other hand, both the studies, including a randomised trial, which reported skills as an outcome in the integrated teaching group found an improvement. Therefore, on balance, there is weak evidence that standalone courses improve appraisal skills and good evidence, including evidence from a randomised trial, that the integrated approach leads to gains in appraisal skills.

Do attitudes change?
Six studies assessed change in attitudes, three each in both teaching groups (fig 1). In the standalone teaching group the three studies, including a randomised trial, did not find a change in attitudes. In the integrated teaching group, however, all studies, including one randomised trial, found an improvement in attitudes. Therefore there is compelling evidence that teaching integrated into clinical practice changes attitudes about the role of EBM or critical literature appraisal in medicine, and a standalone approach does not.

Does behaviour change?
Fourteen studies assessed the outcome of behavioural change after EBM or critical appraisal teaching, including four randomised trials, two in each teaching group (fig 1). The two randomised trials in the standalone group found no change in behaviour, and both randomised trials in the integrated teaching group observed an improvement in behaviour. These findings from the randomised evidence were found to be consistent with the findings of the non-randomised studies, with four of seven studies in the standalone group not showing a change in behaviour and all three non-randomised studies in the integrated teaching group showing benefit. The improvements noted in behaviour included changes in reading habits and choice of information resources, as well as substantial outcomes such as changes in management of patients and guidelines.

Do patients' health outcomes improve?
None of the studies evaluated health outcomes. As the integrated teaching approach showed that it was possible to change behaviour, however, this holds the potential for improving health outcomes.

Discussion

To our knowledge, a comparison of the effects of standalone versus integrated teaching in critical appraisal skills and EBM has not been done before. In addition to not making the distinction between standalone and integrated courses,^2-4 several existing reviews have generally considered undergraduates and postgraduates together. There is empirical evidence, however, that the outcomes of teaching EBM markedly differ between undergraduates and postgraduates, with smaller gains in knowledge among the postgraduates.¹ Moreover, adult learning theory suggests that the determinants of learning in the two groups are different, with postgraduate learning tending to be driven by self motivation and relevance to clinical practice, whereas undergraduate learning is generally driven by external factors such as curriculum and examinations.⁵ This suggests that effectiveness of educational interventions in teaching critical appraisal skills and EBM should be evaluated separately for postgraduate and continuing education, which we have done.

Studies examining the effectiveness of educational interventions may suffer from various weaknesses. Even a randomised controlled study, which is generally regarded as the optimum method for settling questions of effectiveness, is not immune to many of these weaknesses. These weaknesses include difficulty with standardising the educational intervention(s), contamination between the two arms of a study, inability to blind the study participants and the teachers from the educational intervention(s) leading to selective cointervention, and finally difficulty with measuring outcomes due to the lack of valid and reliable assessment tools. Some of these factors make randomised trials unfeasible in educational settings, thus necessitating other designs such as non-randomised controlled and before and after studies. We included all three designs in our review.

We have shown that while standalone teaching and integrated teaching are both effective in improving the knowledge base, it is clinically integrated teaching of EBM that is likely to bring about changes in skills, attitudes, and behaviour (fig 2).

View larger version (65K): [in this window] [in a new window]   Fig 2 Reasons why integrated teaching may achieve better outcomes than standalone teaching

 

The purpose of EBM is to integrate best research evidence with clinical skills and patients' values and preferences.⁶ Teaching EBM should not only equip practitioners with knowledge and skills but also foster their attitudes and encourage the practice of EBM. This is because the ultimate aim of improving care could not be achieved with changes in knowledge and skills alone—it would also require changes in attitudes and behaviour. Critical appraisal and EBM teaching that is integrated into clinical practice seems more effective in improving such substantial outcomes including behavioural changes. Teachers of critical appraisal and EBM should aim to bring teaching out of classrooms into the clinic, but this will require a greater effort.

Summary points Critical literature appraisal and evidence based medicine (EBM) can be taught through standalone courses or through instructional methods that incorporate teaching into routine clinical care Several randomised and non-randomised studies have evaluated the effects of teaching EBM to postgraduates Both standalone courses and integrated teaching improve knowledge Improvements in skills, attitudes, and behaviour, however, come about when teaching is integrated into clinical practice; standalone courses bring about no change It is important to incorporate EBM teaching into clinical practice, but this would require a sustained effort well beyond standalone instruction

---

This is an abridged version; the full version can be found on bmj.com

Contributors: See bmj.com

Funding: None.

Competing interests: AC and KSK have a grant from West Midlands Deanery to teach EBM to specialist registrars in the region, as well as a European Union Grant (LSE031068WM2) to promote EBM among small to medium size enterprises that supply the NHS.

Ethical approval: Not required.

References

1. Norman GR, Shannon SI. Effectiveness of instruction in critical appraisal (evidence-based medicine) skills: a critical appraisal. CMAJ 1998;158: 177-81.[Abstract]
2. Parkes J, Hyde C, Deeks J, Milne R. Teaching critical appraisal skills in health care settings. Cochrane Database Syst Rev 2001;(3): CD001270.
3. Taylor R, Reeves B, Ewings P, Binns S, Keast J, Mears R. A systematic review of the effectiveness of critical appraisal skills training for clinicians. Med Educ 2000;34: 120-5.[CrossRef][Web of Science][Medline]
4. Coomarasamy A, Taylor R, Khan KS. A systematic review of postgraduate teaching in evidence-based medicine and critical appraisal. Med Teach 2003;25: 77-81.[CrossRef][Web of Science][Medline]
5. Knowles MS, Downie CM, Basford P. Teaching and assessing in clinical practice. London: University of Greenwich, 1998: 23-38.
6. Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practice and teach EBM. Edinburgh: Churchill Livingstone, 2000.

(Accepted 23 August 2004)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Articles

Evidence based medicine and the medical curriculum

Paul Glasziou, Amanda Burls, and Ruth Gilbert
BMJ 2008 337: a1253. [Extract] [Full Text]

Evidence based teaching works best at the bedside
BMJ 2004 329: 0. [Full Text]

From optimism to hubris

Jane Smith
BMJ 2004 329: 0. [Extract] [Full Text] [PDF]

What has evidence based medicine done for us?

Sharon E Straus and Giselle Jones
BMJ 2004 329: 987-988. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

• Haden, N. K., Hendricson, W. D., Killip, J. W., O'Neill, P. N., Reed, M. J., Weinstein, G., Williams, J. N., Valachovic, R. W. (2009). Developing Dental Faculty for the Future: ADEA/AAL Institute for Teaching and Learning, 2006-09. J Dent Educ 73: 1320-1335 [Abstract] [Full text]  
• Schreiber, J., Stern, P., Marchetti, G., Provident, I. (2009). Strategies to Promote Evidence-Based Practice in Pediatric Physical Therapy: A Formative Evaluation Pilot Project. ptjournal 89: 918-933 [Abstract] [Full text]  
• De Vito, C., Nobile, C. G., Furnari, G., Pavia, M., De Giusti, M., Angelillo, I. F., Villari, P. (2009). Physicians' knowledge, attitudes and professional use of RCTs and meta-analyses: A cross-sectional survey. Eur J Public Health 19: 297-302 [Abstract] [Full text]  
• Wilder, R. S., Iacopino, A. M., Feldman, C. A., Guthmiller, J., Linfante, J., Lavigne, S., Paquette, D. (2009). Periodontal-Systemic Disease Education in U.S. and Canadian Dental Schools. J Dent Educ 73: 38-52 [Abstract] [Full text]  
• Agrawal, S., Szatmari, P., Hanson, M. (2008). Teaching Evidence-Based Psychiatry: Integrating and Aligning the Formal and Hidden Curricula. Acad. Psychiatry 32: 470-474 [Abstract] [Full text]  
• Feinstein, R. E., Rothberg, B., Weiner, N., Savin, D. M. (2008). University of Colorado Department of Psychiatry Evidence-Based Medicine Educational Project. Acad. Psychiatry 32: 525-530 [Full text]  
• Das, K., Malick, S., Khan, K. S (2008). Tips for teaching evidence-based medicine in a clinical setting: lessons from adult learning theory. Part one. JRSM 101: 493-500 [Abstract] [Full text]  
• Glasziou, P., Burls, A., Gilbert, R. (2008). Evidence based medicine and the medical curriculum. BMJ 337: a1253-a1253 [Full text]  
• Kok, R., Hoving, J. L., Verbeek, J. H., Schaafsma, F. G., Smits, P. B. A., van Dijk, F. J. H. (2008). Evaluation of a workshop on evidence-based medicine for social insurance physicians. Occup Med (Lond) 58: 83-87 [Abstract] [Full text]  
• Hendricson, W. D., Anderson, E., Andrieu, S. C., Chadwick, D. G., Cole, J. R., George, M. C., Glickman, G. N., Glover, J. F., Goldberg, J. S., Haden, N. K., Kalkwarf, K. L., Meyerowitz, C., Neumann, L. M., Pyle, M., Tedesco, L. A., Valachovic, R. W., Weaver, R. G., Winder, R. L., Young, S. K. (2007). Does Faculty Development Enhance Teaching Effectiveness?. J Dent Educ 71: 1513-1533 [Abstract] [Full text]  
• Windish, D. M., Huot, S. J., Green, M. L. (2007). Medicine Residents' Understanding of the Biostatistics and Results in the Medical Literature. JAMA 298: 1010-1022 [Abstract] [Full text]  
• Faggion, C. M. Jr., Tu, Y.-K. (2007). Evidence-Based Dentistry: A Model for Clinical Practice. J Dent Educ 71: 825-831 [Abstract] [Full text]  
• Iacopino, A. M. (2007). The Influence of "New Science" on Dental Education: Current Concepts, Trends, and Models for the Future. J Dent Educ 71: 450-462 [Abstract] [Full text]  
• Chapman, R. L. (2007). Educational Perspectives: Strategies for Teaching and Practicing Evidence-based Neonatology. NeoReviews 8: e105-e109 [Full text]  
• Shaneyfelt, T., Baum, K. D., Bell, D., Feldstein, D., Houston, T. K., Kaatz, S., Whelan, C., Green, M. (2006). Instruments for evaluating education in evidence-based practice: a systematic review.. JAMA 296: 1116-1127 [Abstract] [Full text]  
• Waldman, M. H. (2006). Evidence-based medicine: how to translate research into patient care.. J. Am. Podiatr. Med. Assoc. 96: 374-377 [Abstract] [Full text]  
• Ciliska, D. (2006). Evidence-based nursing: how far have we come? What's next?. Evid. Based Nurs. 9: 38-40 [Full text]  
• Scott Richardson, W, Dowding, D. (2005). Teaching evidence-based practice on foot. Evid. Based Nurs. 8: 100-103 [Full text]  
• Richardson, W S. (2005). Teaching evidence-based practice on foot. Evid. Based Med. 10: 98-101 [Full text]  
• Glasziou, P., Haynes, B. (2005). The paths from research to improved health outcomes. Evid. Based Nurs. 8: 36-38 [Full text]  
• Pennington, L., Roddam, H., Burton, C., Russell, I., Russell, D. (2005). Promoting research use in speech and language therapy: a cluster randomized controlled trial to compare the clinical effectiveness and costs of two training strategies. Clin Rehabil 19: 387-397 [Abstract]  
• Glasziou, P., Haynes, B. (2005). The paths from research to improved health outcomes. Evid. Based Med. 10: 4-7 [Full text]  
• Straus, S. E, Jones, G. (2004). What has evidence based medicine done for us?. BMJ 329: 987-988 [Full text]  
• Del Mar, C., Glasziou, P., Mayer, D. (2004). Teaching evidence based medicine. BMJ 329: 989-990 [Full text]  

Rapid Responses:

Read all Rapid Responses

Single Patient Based Medicine “and” EBM improve medical practice.

Sergio Stagnaro
bmj.com, 29 Oct 2004 [Full text]

Teaching integrated into clinical practice for other subjects should work just as well!

Dukaydah van der Berg
bmj.com, 2 Nov 2004 [Full text]

Not a true comparison of teaching methods

Anna L Pozo
bmj.com, 10 Nov 2004 [Full text]

More research required for integrated teaching of evidence based medicine.

Jeremy Grimshaw, et al.
bmj.com, 7 Dec 2004 [Full text]

Response to letters: standalone versus integrated teaching of EBM

Arri Coomarasamy, et al.
bmj.com, 25 Jan 2005 [Full text]

Some methodological defects

Reza Yousefi Nooraie, et al.
bmj.com, 8 Mar 2005 [Full text]

Online poll

Find out more >>