BMJ  2007;335:333-335 (18 August), doi:10.1136/bmj.39253.544688.94

Analysis

Medical education research remains the poor relation

Department of General Practice and Primary Care, King's College London School of Medicine, London SE11 6SP

Research into medical education is stagnating and urgently needs the resources to become more rigorous and relevant say Mathew Todres, Anne Stephenson, and Roger Jones

The requirement that clinical practice should be based on the best available evidence has been paralleled by calls for medical education to become more evidence based.^{1 2 3} This has resulted, among other initiatives, in the establishment of the Best Evidence for Medical Education (BEME) Collaboration⁴ and the Campbell Collaboration, an off-shoot of the Cochrane Collaboration. The BEME initiative includes dissemination of best evidence to support medical education and the encouragement of a culture capable of nurturing more rigorous and better funded research.

Evidence from the United States suggests such nurturing is much needed. In 2004, Carline analysed reports of medical education research in two major North American journals (Academic Medicine and Teaching and Learning in Medicine) and found that only a minority of studies were supported by external research grants.³ She was critical about the quality, rigour, and generalisability of most of these studies. Her concerns were echoed last year by Chen and colleagues,⁵ who advocated moving the focus of medical education research from learners to patient oriented clinical outcomes, thus increasing the relevance and its likely attractiveness to funders. A review of 290 medical education studies published during 2002 and 2003 found that only one quarter had received external funding; the median amount of funding obtained was $15 000 (£7700; 11 500) with an interquartile range of $5000 to $66 500.⁶ Private foundations, as opposed to federal institutions, were the most common source of these research grants.

Recent medical education research

We were unable to find any recent information about the state of published medical education research in the UK and Europe. We therefore reviewed research published in 2004 and 2005 in two general medical journals, the BMJ and Lancet, and the two leading medical education journals, Medical Education and Medical Teacher. We included only primary research papers and secondary research studies (systematic reviews and meta-analyses).

During 2004 and 2005, none of the 390 research papers published in the Lancet was in the field of medical education. Only 11 of the 399 papers published in the BMJ related to medical education. We combined these with the research papers in Medical Education (207) and Medical Teacher (169) for further analysis.

Research topics, methods, and funding

Most of the research papers used observational, cross sectional survey designs, and less than 10% reported longitudinal cohort and before and after studies (table). Of the 210 (54%) studies that used questionnaires, 178 (85%) provided no details of the validation of survey instruments. Very few papers reported studies using experimental designs, with case-control studies and randomised controlled trials each accounting for less than 3% of the sample. We did not find any meta-analyses.

View this table: [in this window] [in a new window]   Table 1  Details of medical education research published in three journals during 2004-5

 
The research focused predominantly on the undergraduate medical curriculum, with the remaining papers dealing mostly with continuing medical education and postgraduate medical training. Table 2 shows the range of research topics. Student examinations and assessment, curriculum design, professional development, learner characteristics, and teaching methods accounted for 70% of studies. We found little research on potentially important topics such as selecting students for medical school (12 studies), patient issues in medical education (eight), and career choices for medical graduates (four). Most of the research was conducted in the UK, continental Europe, and North America.

View this table: [in this window] [in a new window]   Table 2  Topics for reported studies in medical education, 2004-5

 
The research was largely unfunded. Only 30% (116) of the papers stated that the study had external funding; 47 had internal funding and 224 gave no information on funding, which we assumed meant that there was none. Less than half of these studies (170) were collaborative ventures between two or more institutions. Collaborative studies were more likely to receive external funding than non-collaborative studies (53% v 47%; ²=4.44, P<0.05), and papers published in Medical Education were more likely to be externally funded than those in Medical Teacher (39% v 20%; ²=17.85, P<0.001). We were unable to identify any other associations between publication type, study design, and funding source.

Randomised controlled trials

We identified 10 randomised controlled trials.^{7 8 9 10 11 12 13 14 15 16} Most were published in Medical Education and most concerned undergraduate medical education. These studies largely reported comparisons between different methods of teaching and used a range of assessments and end points. Many of them fell short of the criteria developed by the International Committee of Medical Journal Editors for reporting randomised controlled trials. Several did not include a clear a priori hypothesis, accurately specified end points, or a sample size calculation. Type II statistical errors were common, and CONSORT diagrams indicating the flow of subjects in the control and experimental groups were generally absent.

Problems for research

This review suggests that research into medical education has not moved on, in terms of funding and methods, over the past five to ten years despite repeated initiatives to inject greater methodological rigour and to find better ways of funding studies.

Whether the absence of medical education research from the Lancet and the small number of studies published in the BMJ are causes for concern is debatable. It might be argued that the research should appear in specialist medical education research journals and be read by medical educationalists, but research on topics such as examinations, student selection, predictors of academic success, and professional trajectories in medicine is of wider relevance and deserves larger audiences. The publications in Medical Education and Medical Teacher reflect slow progress in terms of developing methods and attracting funding, with observational designs and unfunded studies still being the rule.

The research landscape in medical education is reminiscent of primary care and health services research 20 years ago, when we lacked a cadre of trained researchers, used primitive research methods, and struggled for funding. The fortunes of applied medical researchers began to improve when funders began to understand the importance of a firm evidence base for clinical care and the delivery of services. It is unclear whether the funders of medical education (the Department for Education and Skills, through the Higher Education Funding Councils) or its providers (the universities and medical schools) are convinced of the importance of a carefully built evidence base to ensure best practice. Even when medical education research is funded, the source of funding is often a charity or foundation rather than a publicly supported funding council. Although several large UK medical research charities provide funding for educational training fellowships, only the Arthritis Research Campaign supports fellowships in medical education research.

The situation is made more difficult by the lack of appreciation of research into medical education in our medical schools. One of the effects of the research assessment exercise (used to allocate research grants in the UK) has been to prioritise the laboratory based sciences over behavioural and applied sciences and research over education, with commensurate shifts in staffing, funding, and academic kudos. This has further disabled medical educators, who are now working harder than ever to keep up, often in the context of high student numbers and increasing bureaucratic complexity. One positive development, however, is that the 2008 assessment exercise will include medical education research.

The medical education research community needs to mobilise support for its mission to provide the best education for medical students. This means thinking of imaginative ways to create a critical mass of educational researchers so that cross-centre, inter-institutional, and multidisciplinary studies can be conducted. Such studies are more likely to produce generalisable results than current research. When these studies can be directly linked to meaningful outcomes, such as the quality of care provided by qualified doctors, they will become more attractive to funders. Critical topics in medical education, such as criteria for selecting medical students, predictors of success and failure at medical school, the development of a mature professional outlook and of personal and interpersonal skills appropriate to various branches of medicine, are all amenable to high quality research.

Undergraduate and postgraduate curriculum reforms over the past 20 years have resulted in major changes in the way that our students and postgraduates are taught, often on the basis of nothing more than pragmatism, fashion, and whim. Not all of these have been good for medical education or for the doctors that it produces—unwieldy student numbers, poorly developed web-based learning introduced as a substitute for direct personal contact, and an obsession with vertical and horizontal integration that led to the destruction of valuable free-standing courses in subjects such as pathology and therapeutics. We must ensure future changes will be beneficial.

Summary points Medical education research lacks methodological rigour and support from funding councils Most studies focus on the undergraduate medical curriculum Important topics such as patient issues in medical education, selection of students, and career choices are under-researched Multicentre studies using good experimental designs are needed

---

Contributors and sources: MT has been a research associate in medical education since July 2006. Before that he worked as a project manager and research assistant on an action research project in social enterprise development. AS has been a medical educationalist for 25 years with a particular interest is in the professional development of medical students and medical teachers. RJ has helped to develop undergraduate medical curriculums and assessment methods for over 20 years, and has a particular interest in factors which predict success and failure in undergraduate medical education. RJ and AS had the original idea for the systematic review. MT conducted the review. RJ evaluated the randomised controlled trials. RJ and AS supervised the review process, and all of us contributed to writing the paper. RJ is the guarantor.

Competing interests: None declared.

Provenance and peer review: Not commissioned; externally peer reviewed.

References

1. Bligh J, Anderson MB. Medical teachers and evidence. Med Educ 2000;24:162-3.
2. Wood D, Bligh J. Medical education comes of age. Med Educ 2000;34:82-3.[CrossRef][ISI][Medline]
3. Carline JD. Funding medical education research: opportunities and issues. Acad Med 2004;79:918-24.[CrossRef][ISI][Medline]
4. Hart I. Best evidence medical education . Med Teacher 1999;21:453-4.[CrossRef]
5. Chen FM, Burstin H, Huntington J. The importance of clinical outcomes in medical education research. Med Educ 2005;39:350-1.[CrossRef][ISI][Medline]
6. Reed DA, Kern DE, Levine RB, Wright SM. Costs and funding for published medical education research. JAMA 2005;294:1052-7.[Abstract/Free Full Text]
7. Tamayo G, Santibanez M, Meanna JJ. Evaluation of a pharmacology educational activity based on a research project: a randomised controlled and blind analysis of medical student perceptions. Med Teach 2005;27:53-60.[CrossRef][ISI][Medline]
8. Bradley P, Oterholt C, Herrin J, Nordheim L, Bjorndal A. Comparison of directed and self-directed learning in evidence-based medicine: a randomised controlled trial. Med Educ 2005;39:1027-35.[CrossRef][ISI][Medline]
9. Westberg K, Sandlund M, Lynoe N. The effect of giving written information in advance on the clinical training of medical students. Med Educ 2005;39:1021-6.[CrossRef][ISI][Medline]
10. Junger J, Schafer S, Roth C, Schellberg D, Friedman Ben-David M, Nikendei C. Effects of basic clinical skills training on objective structured clinical examination performance. Med Educ 2005;39:1015-20.[CrossRef][ISI][Medline]
11. Spickard A, Smithers J, Cordray D, Gigante J, Wofford JL. A randomised trial of an online lecture with and without audio. Med Educ 2004;38:787-90.[CrossRef][ISI][Medline]
12. Ochsendorf FR, Boehncke WH, Boer A, Kaufmann R. Prospective randomised comparison of traditional personal bedside and problem-orientated dermatology courses. Med Educ 2004;38:652-8.[CrossRef][ISI][Medline]
13. Margolis P, Lannon CM, Stuart JM, Fried BJ, Keyes-Elstein L, Moore DE. Practice based education to improve systems for prevention in primary care: randomised trial. BMJ 2004;328:388-94.[Abstract/Free Full Text]
14. Qayumi AK, Kurihara Y, Imai M, Pachev G, Seo H, Hoshino Y, et al. Comparison of computer assisted instruction (CAI) versus traditional textbook methods for training in abdominal examination (Japanese experience). Med Educ 2004;38:1080-8.[CrossRef][ISI][Medline]
15. Schol S, Goehuys J, Notten T, Betz W. Individualised training to improve teaching competence of general practitioner trainers: a randomised controlled trial. Med Educ 2005;39:991-8.[CrossRef][ISI][Medline]
16. Bridgemohan CF, Levy S, Veluz AK, Knight JR. Teaching paediatric residents about learning disorders: use of standardised case discussion versus multimedia computer tutorial. Med Educ 2005:797-806.

(Accepted 4 June 2007)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Relevant Articles

Reasons to be cheerful

Julian Archer
BMJ 2007 335: 414-415. [Extract] [Full Text] [PDF]

Other journals, other means

David Wall and Guy Houghton
BMJ 2007 335: 415. [Extract] [Full Text] [PDF]

Practice based education to improve delivery systems for prevention in primary care: randomised trial

Peter A Margolis, Carole M Lannon, Jayne M Stuart, Bruce J Fried, Lynette Keyes-Elstein, and Donald E Moore, Jr
BMJ 2004 328: 388. [Abstract] [Full Text] [PDF]

This article has been cited by other articles:

• Walsh, K. (2008). An academy of medical educators. JRSM 101: 213-214 [Full text]  
• Wall, D., Houghton, G. (2007). Other journals, other means. BMJ 335: 415-415 [Full text]  
• Archer, J. (2007). Reasons to be cheerful. BMJ 335: 414-415 [Full text]  

Rapid Responses:

Read all Rapid Responses

Reply to Todres, Stephenson and Jones

David W Wall, et al.
bmj.com, 21 Aug 2007 [Full text]

Medical Education: Training First

Muhammad N. Ghayur
bmj.com, 21 Aug 2007 [Full text]

Only half a story!

John E Sandars
bmj.com, 21 Aug 2007 [Full text]

There is reason for optimism

Julian Archer
bmj.com, 22 Aug 2007 [Full text]

MMC: help or hinderance?

Rachel E Owers
bmj.com, 23 Aug 2007 [Full text]

Something needs to be done

Robert K McKinley
bmj.com, 23 Aug 2007 [Full text]

On the nature of medical education research

Samantha Scallan, et al.
bmj.com, 31 Aug 2007 [Full text]

Healthcare education research: Funding and RCTS are both needed

Mark J Newman
bmj.com, 31 Aug 2007 [Full text]

Theoretical rigor? Let's start by making explicit the implicit

Lynn V Knight, et al.
bmj.com, 31 Aug 2007 [Full text]

Pictures are important too

Stewart Lloyd
bmj.com, 10 Sep 2007 [Full text]

Medical Education Research

Mathew Todres, et al.
bmj.com, 11 Sep 2007 [Full text]