General practitioners' self ratings of skills in evidence based medicine: validation studyBMJ 2002; 324 doi: http://dx.doi.org/10.1136/bmj.324.7343.950 (Published 20 April 2002) Cite this as: BMJ 2002;324:950
- Jane M Young, research fellow ()a,
- Paul Glasziou, professor of evidence based practiceb,
- Jeanette E Ward, directorc
- a Needs Assessment and Health Outcomes Unit, Central Sydney Area Health Service, Camperdown, NSW 2050, Australia
- b School of Population Health, University of Queensland, Brisbane, Queensland 4006, Australia
- c Division of Population Health, South Western Sydney Area Health Service, Liverpool, NSW 2170, Australia
- Correspondence to: J Young
- Accepted 15 October 2001
To practise evidence based medicine, clinicians need to understand and use terms such as “relative risk reduction,” “absolute risk reduction,” and “number needed to treat.”1 Self ratings represent one method of assessing competence in these skills. About a third of clinicians claim to understand such terms.2 We evaluated the validity of self ratings and conducted a blinded validation in general practice.
Methods and results
Fifty general practitioners in Sydney, Australia, completed self administered questionnaires,2 in which they rated their understanding of each of seven terms used in evidence based medicine as “Would not be helpful for me to understand,” “I don't understand but would like to,” “I already have some understanding,” and “I understand this and could explain to others.” We considered the last response to represent full understanding (self rating of competence). Participants sealed their responses in an envelope before participating in a structured interview with JY (who was unaware of their self rating), in which they were asked to explain each term as if to a medical student. Unprompted comments were recorded (see box on bmj.com). The study was approved by the Central Sydney Area Health Service Ethics Review Committee.
Three independent experts in evidence based medicine had been asked to identify criteria essential for showing that the participant knew the correct meaning of the term (criterion based assessment; see table on bmj.com). During interviews with general practitioners, JY ticked any criterion met by participants' verbal explanations. To demonstrate competence in understanding number needed to treat, for example, participants had to include in their verbal responses the concept that this represents the number of patients needed to be treated to achieve one good outcome or prevent one bad outcome and that it is the reciprocal of absolute risk reduction.
Participants' verbal explanations almost never met the essential criteria (table). Although self ratings were modest, only one participant's explanation met all essential criteria, and this for only one term, positive predictive value.
We could not calculate sensitivity and specificity of self rated competence for any terms other than positive predictive value as only one respondent met objective criteria for competence. We calculated positive and negative predictive values for each term to assess the probability of competence given a positive or negative self rating. The predictive value of a positive self rating was 8% for positive predictive value but zero for the other six terms (table). As no participants demonstrated competence exceeding their self rating, the predictive value of a negative self rating was 100% for all terms.
Participants' self ratings of their understanding of terms used in evidence based medicine differed from an objective, criterion based assessment. Moreover, participants' comments showed considerable misunderstanding about terms.
Medical education in Australia has largely not prepared general practitioners for evidence based medicine. Remediation is crucial if they are to understand research findings on which clinical practice ought to be based and avoid pitfalls such as “framing effect.”3 Little rigorous research has been conducted to identify effective educational strategies for clinicians.4
It is unclear whether findings from our modest sample also apply to medical practitioners in other settings. Australia's general practitioners are at least as familiar with evidence based medicine as their counterparts in other countries, given recent focus in health policy.5 Our method may have resulted in underperformance by participants, who might have been able to explain these terms to medical students when not under the scrutiny of an academic interviewer. Furthermore, general practitioners may understand these terms less in the abstract but more when they are used in context by a conference speaker or in a research article.
We thank the general practitioners who participated in this study and Jeremy Anderson, Chris Del Mar, and Chris Silagy, who responded to our request to rate criteria.
Contributors: JW and PG were responsible for the study concept. All authors developed the study protocol. JY conducted the study and analysed data. All authors jointly wrote the paper. JY is the guarantor.
Editorial by Woodcock et al
Funding At the time of the fieldwork, JY was employed by Central Sydney Area Health Service. JY is currently supported by National Health and Medical Research Council Public Health (Australia) fellowship No 007024.
Competing interests None declared
See box and additional table on bmj.com