Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicineBMJ 2002; 325 doi: https://doi.org/10.1136/bmj.325.7376.1338 (Published 07 December 2002) Cite this as: BMJ 2002;325:1338
Development of instrument
We derived the content from course topics of a published curriculum, which included basic epidemiological, statistical, and informational concepts relevant to evidence based medicine in the clinical situation (diagnostic tests, therapy, prognosis, harm, and systematic reviews).w1 The five experienced teachers in evidence based medicine—clinicans (ND-B, LF, and RK), clinical epidemiologists (ND-B, RK, H-WH), and a statistician (KW)—drafted the first set of multiple choice questions. We explicitly sought to assess deep learning (the ability to integrate concepts and to apply them in new situations) rather than superficial learning (the ability to recall definitions and facts). When calculations were part of the solution, the answer included the calculation path. Participants had to identify the correct path rather than merely select the correct numerical solution.
We recirculated the questions according to a modified Delphi method and we assessed for unambiguous wording and answers, clinical relevance, and importance for novices in evidence based medicine.w2 Disagreements were resolved by discussion. We piloted the questions for understanding and feasibility in a group of five doctor volunteers not familiar with evidence based medicine and then modified the questions further. At the end of the first stage the instrument consisted of two sets of 15 test questions, with similar content applied to different case scenarios to allow repeated testing.
Validation of instrument
We assessed the difficulty of each question by calculating the proportion of correct answers for each question. We sought to ensure that scores per question by course participants should not fall below 0.1 or go above 0.9, as scores outside these parameters do not tend to provide additional information to distinguish the more knowledgeable participants from less knowledgeable ones. Equivalence of sets 1 and 2 was determined by calculating the intraclass correlation coefficient for the controls (students and experts), who received the test without intervention. We determined the internal consistency of the instrument, a measure of reliability as described in standard psychometric textbooks.w3 w4 We calculated Pearson’s correlation coefficient to assess the relation between an individual item and the instrument as a whole omitting that item (item total correlationw5) to identify items contributing to a low reliability. As a rule of thumb, the correlation of that item with the total score should be above 0.2.w3 In addition, we calculated a standard summary measure for internal consistency, the coefficient α (Cronbach’s α ).w6 We considered a coefficient above 0.7 as a satisfactory level of internal consistency.w6
We assessed the instrument’s ability to discriminate between groups with different expertise in evidence based medicine by assessing experts, students, and course participants. Both sets of the instrument were administered to the expert controls and the student controls. We analysed the following comparisons: experts versus course participants (before test) versus students; course participants before the course (group A versus group B versus group C), using one way analysis of variance. If we found a significant difference, we added a post hoc analysis by Scheffé’s method.
Measurement of educational effect
The 3 day Berlin course on evidence based medicine was developed in 1997, based on the small group educational model developed at McMaster University, Canada and directed towards doctors at all levels of training.w7 It covers previously defined core content in 22 hours of small group sessions and 5 hours of plenary presentations.w1 It uses a problem based learning style with case scenarios of typical patient problems supported by relevant primary studies. We provided hands-on exercises on retrieval of information and refresher courses in basic statistics. Each group was run by two tutors,an experienced methodologist and a doctor with several years’ clinical experience. The sessions introduced doctors to the principles of evidence based medicine (identification and prioritisation of problems, formulation of questions, consideration of clinical decision options) and critical appraisal in the context of the patient scenario and helped them to become skilled users of evidence that had already been appraised.
w1 Kunz R, Fritsche L, Neumayer H-H. Development of quality assurance criteria for continuing education in evidence-based medicine. Z Arztl Fortbild Qualitatssich 2001;95:3712-5.
w2 Greenhalgh T. Consensus research. In: Elwyn G, Greenhalgh T, Macfarlane F, ed. Groups—a guide to small group work in health care management, education and research. Oxford: Radcliffe Medical Press, 2000.
w3 Streiner DL, Norman GR. Health measurement scales. Oxford: University Press, 1995.
w4 Lienert GA, Raatz U. Testaufbau und testanalyse. Weinheim: Psychologie Verlagsunion, 1995.
w5 Nunnally JC. Introduction to psychological measurement. New York: McGraw-Hill, 1970.
w6 Nunnally JC. Psychometric theory. New York: McGraw-Hill, 1978.
w7 The Evidence-Based Medicine Working Group. Users’ guides to the medical literature: a manual for evidence-based clinical practice. Chicago: JAMA and Archives Journals, 2002.
Control group Initial set* Subsequent set Score (SD) No returning both sets Experts Set 1 (30) Set 1 (10) 11.7 (1.6) 20 Set 2 (13) Set 2 (10) 12.5 (1.4) Students Set 1 (10) Set 1 (10) 4.0 (2.3) 20 Set 2 (10) Set 2 (10) 4.5 (2.1)
*Sequence of test sets were reversed year on year, so that group completing initial set 1 went on to complete subsequent set 2 and group completing initial set 2 went on to complete subsequent set 1.
Table B Sequence and number of completed sets of Berlin Questionnaire by participants on three courses in evidence based medicine. Values in parentheses are number completing set unless stated otherwise
Participant group Set before course Score (SD) Set after course Score (SD) No returning both sets Course A, 1999 Set 1 (61) 5.8 (2.8) Set 2 (60) 9.6 (2.4) 42 Course B, 2000 Set 2 (37) 6.9 (2.8) Set 1 (46) 10.2 (2.7) 33 Course C, 2001 Set 2 (46) 6.6 (3.0) Set 1 (70) 10.0 (2.2) 46 Total 144 6.3 (2.9) 176 9.9 (2.4) 121
- The BMJ Christmas appeal 2016-7: Orbis, the sight saversBMJ December 01, 2016, 355 i6425; DOI: https://doi.org/10.1136/bmj.i6425
- A musical about malignancyBMJ November 23, 2016, 355 i6318; DOI: https://doi.org/10.1136/bmj.i6318
- Round-up from Leaders in Healthcare conferenceBMJ November 08, 2016, 355 i5977; DOI: https://doi.org/10.1136/bmj.i5977
- Why can’t the NHS value junior staff the way top companies do?BMJ October 26, 2016, 355 i5740; DOI: https://doi.org/10.1136/bmj.i5740
- Reduce NHS’s reliance on migrants, HEE is toldBMJ October 18, 2016, 354 i5614; DOI: https://doi.org/10.1136/bmj.i5614
- Does a one-day workshop improve clinical facultys comfort and behaviour in practising and teaching evidence-based medicine? A Canadian mixed methods study
- The impact of clinical maturity on competency in evidence-based medicine: a mixed-methods study
- Assessing Evidence-Based Practice Knowledge, Attitudes, Access and Confidence Among Dental Hygiene Educators
- Development of the Biostatistics and Clinical Epidemiology Skills (BACES) assessment for medical residents
- Comparison of Three Evidence-Based Practice Learning Assessment Methods in Dental Curricula
- Evidence-Based Dentistry and Clinical Implementation by Third-Year Dental Students
- Effect of Dental Education on Peruvian Dental Students' Oral Health-Related Attitudes and Behavior
- Tools to assess Evidence-Based Practice behaviour among healthcare professionals
- Validation of an Instrument to Assess Evidence-Based Practice Knowledge, Attitudes, Access, and Confidence in the Dental Environment
- Evaluating the teaching of evidence based medicine: conceptual framework
- Why does the MRCPsych examination need to change?