BMJ 1999;318:1265-1267 ( 8 May )

Education and debate

Guidelines for evaluating papers on educational interventions

Education Group for Guidelines on Evaluation. 

Correspondence to: K Abbasi kabbasi{at}bmj.com

Education is an important part of the work of most doctors, and the BMJ is interested to publish original studies that will be useful to doctors in their educational role. Unfortunately many of the accounts we receive of educational interventions comprise a thin description of the innovation and an evaluation that says little more than that the students liked the innovation. This is not good enough. The standard of papers evaluating educational interventions should be as high as that of any other original papers that we publish.

We recognise, however, that many of the methodologies that are best for evaluating educational innovations are different from the methods with which BMJ readers are familiarfor instance, methods for evaluating new drugs. We thus set up a group of advisers, consisting of people expert in medical education, to produce guidelines that we could use when reviewing original papers that describe educational innovations. This is a first version of those guidelines. The guidelines are intended for authors, editors, reviewers, and readers. We have no doubt that they can be improved, and it might be that we should evolve different guidelines for those different groups.

We are doing three things with these guidelines:

• Publishing them today and inviting comment. The group that produced the guidelines will revise them in the light of the responses we receive;
• Sending them to various groups and individuals interested in evaluating educational innovations and asking them to comment;
• To see if the guidelines work in practice, we plan to use them in reviewing papers describing educational innovations.

Summary points Doctors are increasingly involved in education, and they should benefit from being exposed to research in medical education General medical journals have published little educational research, but the BMJ is interested to publish more The methods used in educational research are often different from those most familiar to readers of general medical journals The journal has therefore worked with education experts to develop guidelines for authors, editors, reviewers, and readers for evaluating studies on educational interventions Two crucial factors in good studies are that the educational rationale behind the intervention is made explicit and that the evaluation is planned in advance The guidelines are being widely circulated and will then be revised before final adoption

	Guidelines for authors, editors, reviewers, and readers to evaluate papers on educational interventions

1 Overview
(a) Is the paper right for the BMJ?

Papers on educational interventions are of several types. Retrospective and descriptive studies that describe a change in a programme or module, or the development of a new teaching method or curriculum, may be of interest to those involved in curriculum management, but are less likely to be of general enough interest for the BMJ.

Well conducted studies examining educational innovations have a better chance of publication. These may take various forms: detailed observational studies, properly conducted questionnaire surveys, or randomised controlled studies. In all cases the design and evaluation criteria listed in section (3) below should be applied.

(b) Does it add anything new and valuable?

The BMJ is most interested in publishing studies that are genuinely original in that nobody has ever done anything like this before. There is also a place for studies that confirm and extend previous studies. The BMJ will be interested in the first few studies that confirm previous studies, particularly if they are methodologically superior to previous work and extend it. But the BMJ is not interested in studies that confirm what has been shown several times beforealbeit in different countries and settings.

(c) Is it suitable for a general readership?

The BMJ is interested in material that will be useful and understandable to a general audience. Papers that are intended primarily for an audience with a specialist interest in education should be published elsewhere. Authors of educational papers that are submitted to the BMJ should avoid educational jargon; at the very least, jargon should be explained simply and fully in the textand perhaps in a glossary.

(d) Is it readable?

It is important that the paper is presented in a logical fashion and written in a coherent, readable style. The tables and diagrams should be well presented, useful, and relevant.

2 Theoretical considerations
(a) Are the aims and objectives clearly stated?

The aims and objectives of the study should be clearly stated. The educational rationale, context of the study, and methodology should relate to the aims and objectives. The research techniques used must be appropriate to answer the question(s) posed in the aims, and to achieve the learning objectives.

Summary of guidelines for evaluating papers on educational interventions Checklist 1 Overview (c) Are the methods used for recruitment described in enough detail? (a) Is the paper right for the BMJ? (d) Was the evaluation method planned in advance and linked to the aims of the study? (b) Does it add anything new and valuable? (e) Is the evaluation tool described in enough detail? (c) Is it suitable for a general readership? (f) Are the results meaningful? (d) Is it readable? 4 Discussion 2 Theoretical considerations (a) Is it well structured? (a) Are the aims and objectives clearly stated? (b) Does it discuss the strengths and weaknesses of the study in relation to other studies? (b) Is the educational rationale explicit? (c) Does it discuss the meaning and implications of the results? (c) Is the intervention described in context? (d) Does it discuss the need for further work? 3 Study presentation and design (a) Is the method described in enough detail? (b) Does the study allow the questions posed to be answered?

(b) Is the educational rationale explicit?

The educational rationale behind the innovation should be explicit, and it should be obvious from the paper that the study is founded on the application of theoretical principles. An adequate review of the literature should be given to support the basis of the study.

(c) Is the intervention described in context?

The context of educational change is important and may have a direct bearing on the implementation of change. The paper should describe the context of the intervention in terms of healthcare delivery systems, political policy, and external drives to encourage change where these have been important. In addition, the study should take into account the study population and the stage of educational development, and it should describe the relevant details of local issues such as the individual course or module, its place within the curriculum, and the physical environment in which the study took place.

3 Study presentation and design
(a) Is the method described in enough detail?

As educational interventions are often specific to the context in which they take place, a large amount of background detail may be necessary to familiarise others. A balance is needed between enough detail to allow scrutiny and reproducibility of the intervention, and information overload. It is always possible to publish more detailed information in the eBMJ.

(b) Does the study allow the questions posed to be answered?

The aims of the intervention should be reflected in the aims of the research and then in the methodology selected.

(c) Are the methods used for recruitment described in enough detail?

The method of recruitment needs justification. If control groups are used, the process of selecting controls should be fully described and rigorous. The usual policy of the BMJ is to publish only trials in which controls are randomly selected unless there is a convincing reason why randomisation is not possible.¹ Randomised trials should conform with the CONSORT criteria. ^{2 3} Purposive sampling may be more informative than attempts at randomisation and controls, which are difficult to achieve in adult education.

(d) Was the evaluation method planned in advance and linked to the aims of the study?

Evaluations should not be an afterthought. Every evaluation has a purpose specific to the research question and the context. Researchers planning an intervention should have designed the evaluation at the outset to answer their specific questions.

(e) Is the evaluation tool described in enough detail?

An evaluation may be valid and useful within the context in which it took place, without meeting the criteria required for research. To achieve generalisability or reproducibility, there needs to be a higher level of rigour. Attempts should be made to correlate reported behaviour changes after an intervention with more objective measures such as referral rates and with investigation patterns.

(f) Are the results meaningful?

Educational interventions are often difficult to analyse because multiple variables are involved and because there may not be only one explanation for the results. The results need to be presented in sufficient detail to be meaningful, and the statistical analysis should be appropriate for the study design.

4 Discussion
(a) Is it well structured?

The BMJ is proposing to introduce structured discussions,⁴ and these should be used in educational articles as in any others. The discussion should begin with a sentence on the principal finding, followed by a thorough examination of the strengths and weaknesses of the study itself.

(b) Does it discuss the strengths and the weaknesses of the study in relation to other studies?

Strengths and weaknesses should then be discussed in relation to previous studies. Any differences in results, and why different conclusions have been reached, should particularly be emphasised.

(c) Does it discuss the meaning and implications of the results?

Next, the "meaning" of the study in terms of possible mechanisms, and implications for clinicians or policy makers, needs to be explored but should not be overstated.

(d) Does it discuss the need for further work?

Finally, questions that remain unanswered and what future work is needed should be discussed without being speculative.

	Acknowledgments

Contributors: Luisa Dillner, Linda Hutchinson, Jean Kerr, Sam Leinster, Kwee Matheson, Stewart Peterson, Mark Rake, Tessa Richards, Richard Smith, and Diana Wood were on the advisory group that formulated the new guidelines and they, along with Peter Cantillon, reviewed an early draft, which was prepared by Kamran Abbasi, Linda Hutchinson, Richard Smith, and Diana Wood. The guidelines were edited by Kamran Abbasi and Richard Smith. Richard Smith is guarantor.

	Footnotes

Members of the group are listed at the end of the article

	References

1. Altman DG. Randomisation. BMJ 1991; 292: 810-812.
2. Altman DG. Better reporting of randomised trials: the CONSORT statement. BMJ 1996; 313: 570-571[Free Full Text].
3. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomised controlled trials: the CONSORT statement. JAMA 1996; 276: 637-639[Medline].
4. Docherty M, Smith R. The case for structuring the discussion of scientific papers. BMJ 1999; 318: 1224-1225[Free Full Text].

---

© BMJ 1999
CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Relevant Articles

Effectiveness of the diabetes education and self management for ongoing and newly diagnosed (DESMOND) programme for people with newly diagnosed type 2 diabetes: cluster randomised controlled trial

M J Davies, S Heller, T C Skinner, M J Campbell, M E Carey, S Cradock, H M Dallosso, H Daly, Y Doherty, S Eaton, C Fox, L Oliver, K Rantell, G Rayman, K Khunti on behalf of the Diabetes Education and Self Management for Ongoing and Newly Diagnosed Collaborative
BMJ 2008 336: 491-495. [Abstract] [Full Text] [PDF]

"I don't know": the three most important words in education
BMJ 1999 318: 0. [Full Text] [PDF]

"I don't know": the three most important words in education
BMJ 1999 318: 0. [Full Text]

Time for evidence based medical education

Stewart Petersen
BMJ 1999 318: 1223-1224. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

• Davies, M J, Heller, S, Skinner, T C, Campbell, M J, Carey, M E, Cradock, S, Dallosso, H M, Daly, H, Doherty, Y, Eaton, S, Fox, C, Oliver, L, Rantell, K, Rayman, G, Khunti, K, on behalf of the Diabetes Education and Self Manag, (2008). Effectiveness of the diabetes education and self management for ongoing and newly diagnosed (DESMOND) programme for people with newly diagnosed type 2 diabetes: cluster randomised controlled trial. BMJ 336: 491-495 [Abstract] [Full text]  
• Reed, D. A., Cook, D. A., Beckman, T. J., Levine, R. B., Kern, D. E., Wright, S. M. (2007). Association Between Funding and Quality of Published Medical Education Research. JAMA 298: 1002-1009 [Abstract] [Full text]  
• Baernstein, A., Liss, H. K., Carney, P. A., Elmore, J. G. (2007). Trends in Study Methods Used in Undergraduate Medical Education Research, 1969-2007. JAMA 298: 1038-1045 [Abstract] [Full text]  
• Siriwardena, A., Fairchild, P, Gibson, S, Sach, T, Dewey, M (2007). Investigation of the effect of a countywide protected learning time scheme on prescribing rates of ramipril: interrupted time series study. Fam Pract 24: 26-33 [Abstract] [Full text]  
• Reed, D., Price, E. G., Windish, D. M., Wright, S. M., Gozu, A., Hsu, E. B., Beach, M. C., Kern, D., Bass, E. B. (2005). Challenges in Systematic Reviews of Educational Intervention Studies. ANN INTERN MED 142: 1080-1089 [Abstract] [Full text]  
• Wass, V., Richards, T., Cantillon, P. (2003). Monitoring the medical education revolution. BMJ 327: 1362-1362 [Full text]  
• Cottrell, D. (2003). Raising the standards of educational evaluation and research. Psychiatr. Bull. 27: 1-2 [Full text]  
• Green, M. L. (2001). Identifying, Appraising, and Implementing Medical Education Curricula: A Guide for Medical Educators. ANN INTERN MED 135: 889-896 [Abstract] [Full text]  
• EDEN, T. (2000). Evidence based medicine. Arch. Dis. Child. 82: 275-277 [Full text]  

Rapid Responses:

Read all Rapid Responses

Practical and Helpful

Joella Gipson
bmj.com, 17 Feb 2004 [Full text]