Introduction

Top Abstract Introduction Participants and methods Results Discussion References

The patterns of health need in youth have changed noticeably over the past three decades. Studies in the United Kingdom, North America, and Australia have shown that young people experience barriers to health services.^1-5 With the increase in a range of youth health problems, such as depression, eating disorders, drug and alcohol use, unplanned pregnancy, chronic illness, and suicide, there is a need to improve the accessibility and quality of health services to youth. ^{3 6}

In the Australian healthcare system general practitioners provide the most accessible primary health care for adolescents.⁷ Yet Veit et al surveyed 1000 Victorian general practitioners and found that 80% reported inadequate undergraduate training in consultation skills and psychosocial diseases in adolescents and 87% wanted continuing medical education in these areas. ^{4 8} These findings agreed with comparable overseas studies.^9-11

Evidence based strategies in helping doctors learn and change practice are at the forefront of the design of continuing medical education.^12-14 In response to the identified gap in training an evidence based educational intervention was designed to improve the knowledge, skill, and self perceived competency of general practitioners in adolescent health. We conducted a randomised controlled trial to evaluate the intervention, with follow up at seven and 13 months after the baseline assessment.

	Participants and methods

Top Abstract Introduction Participants and methods Results Discussion References

The divisions of general practice are regional organisations that survey the needs of, and provide education for, general practitioners in their zone. There are 15 divisions in metropolitan Melbourne. Advertisements inviting participation in our trial were placed in 14 of the 15 divisional and state college newsletters and mailed individually to all division members. The course was free, and continuing medical education points were available. Respondents were sent details of the intervention and the evaluation protocol and asked to return a signed consent form. Divisions and doctors were excluded if they had previously received a course in adolescent health from this institution.

Randomisation
Consenting doctors were grouped into eight geographical clusters by practice location to minimise contamination and to maximise efficiency of the delivery of the intervention. Clusters (classes) of similar size were randomised to intervention or control by an independent researcher.

Intervention
The box details the objectives, content, and instructional design of the multifaceted intervention. A panel comprising young people, general practitioners, college education and quality assurance staff, adolescent health experts, and a state youth and family government officer gave advice on the design.¹⁵ The curriculum included evidence based primary and secondary educational strategies such as role play with feedback, modeling practice with opinion leaders, and the use of checklists. ^{12 16} The six week programme was delivered concurrently by LS, starting one month after baseline testing (see figure on website).

Measures
Table 1 summarises the instruments used in the evaluation. Parallel strategies of objective and self reported ratings of knowledge, skill, and competency were used to ensure findings were consistent. ^{17 18} Participants' satisfaction with the course and their self reported change in practice were evaluated at 13 months. Any other training or education obtained in adolescent health or related areas were noted.

Clinical skills
Seven female drama students were trained to simulate a depressed 15 year old exhibiting health risk behaviour. Case details and performances were standardised according to published protocols^19-21 and varied for each testing period. Doctors were given 30 minutes to interview the patient in a consulting room at this institution. An unattended camera recorded the consultation on videotape.

Self perceived competency
Two questionnaires were developed for the doctors to rate both their comfort and their knowledge or skill with process issues, including the clinical approach to adolescents and their families and with substantive issues of depression, suicide risk assessment, alcohol and drug issues, eating disorders, sexual history taking, and sexual abuse. Doctors also rated their consultation with the standardised patient on a validated chart,²¹ itemising their self perceived knowledge and skill.

Knowledge
Knowledge was assessed with short answer and multiple choice items developed to reflect the workshop topics. The items were pretested and refined for contextual and content validity. The course tutor, blind to group status, awarded a summary score.

Analysis
Statistical analysis was performed with STATA (Stata, Texas), with the individual as the unit of analysis. Factor analysis with varimax rotation was used to identify two domains within the comfort and self perceived knowledge or skill items: process and substantive issues. The internal consistency for all scales was estimated using Crohnbach's . Reproducibility within and between raters was estimated with one way analysis of variance as was the intraclass correlation of baseline score within each teaching group.

	Results

Top Abstract Introduction Participants and methods Results Discussion References

Participants
Newsletters and mailed advertisements to 2415 general practitioners resulted in 264 expressions of interest. Overall, 139 doctors gave written consent to be randomised. Attrition after notification of study status left 55 (73%) doctors in the intervention group and 53 (83%) in the control group, with an average of 13.5 (12 to 15) doctors in each class.

Compliance
One doctor dropped out of the intervention group. Overall, 44 doctors attended all six tutorials, eight missed one, and two missed three. In total, 103 of 108 (95%) of participants at baseline completed the entire evaluation protocol (see website).

Measures
The evaluation scales showed satisfactory internal consistency and low association with class membership (table 1). Satisfactory interrater agreement was achieved on the competency scale (n=70, r=0.70). The intrarater consistency for both medical and non-medical raters was also satisfactory (n=20, r=0.80 and 0.91 respectively).

Effect of the intervention
Table 3 describes the baseline measures and the effect of the intervention at the seven month follow up. All analyses were adjusted for age, gender, languages other than English, average weekly hours of consulting, practice type, and college examinations. Doctors reporting education in related areas during follow up (67% control (34 of 51), 41% intervention (22 of 54)) were characterised. The difference analysis was adjusted for this extraneous training and baseline score, although the extraneous training did not affect any outcomes. The study groups were similar on all measures at baseline. The intervention group showed significantly greater improvements than the control group at the seven month follow up in all outcomes except the rapport rating by the standardised patients.

Follow up of the intervention group at 13 months
The intervention effect was sustained in most measures and further improved in the independent rater's assessment of competence (table 4). The crude rating of the confidentiality discussion by the standardised patients deteriorated at the 13 month assessment but was significantly greater than baseline. Overall, 98% of the participants reported a change in practice, which they attributed to the intervention.

	Discussion

Top Abstract Introduction Participants and methods Results Discussion References

A course in adolescent health for six sessions designed with evidence based strategies in doctor education brought substantial gains in knowledge, skills, and self perceived competency of the intervention group of doctors compared with the control group, except for the rapport and satisfaction rating by the standardised patients. The changes were generally sustained over 12 months and further improved in the independent observer's rating of competence. Almost all participants reported a change in actual practice since the intervention.

These results are better than reported in a review of 99 randomised controlled trials to evaluate continuing medical education published from 1974-95.¹² Although over 60% had positive outcomes they were small to moderate and usually in only one or two outcome measures. In keeping with the recommendations of this review we adapted a rigorous design, clearly defined our target population, and used multiple methods for evaluating competence. Perhaps more importantly the intervention design incorporated three further elements: the use of evidence based educational strategies, a comprehensive preliminary needs analysis, and the content validity of the curriculum ensured by the involvement of both young people and doctors.

The participants clearly represented a highly motivated group of doctors. This self selection bias was unavoidable but reflected the reality that only interested doctors would desire special skills in this domain and conforms to the adult learning principle of providing education where there is a self perceived need and desire for training. ^{12 26 27} We therefore established that the intervention is effective with motivated doctors. It is generally accepted that doctors with an interest in a topic would already have high levels of knowledge and skill, with little scope for improvement. This was not the case in our study. Baseline measures were often low and improvements were large, confirming the need for professional development in adolescent health. The retention rate was excellent and possibly due, in part, to the role of a doctor in the design of the programme, in recruitment, and in tutoring.

Doubt remains as to whether improved competency in a controlled test setting translates to improved performance in clinical practice.²⁸ High competency ratings are not necessarily associated with high performance, but low competency is usually associated with low performance. ^{16 29 30}

The rapport and satisfaction rating by the standardised patients was the only outcome measure apparently unresponsive to the intervention. Actors' ratings and character portrayal were standardised, and gender bias was controlled by using only actresses. Even with these precautions three actresses scored differently from the rest, one had fewer physician encounters, and the subjective nature of the rating scale probably contributed to large individual variation. A trend towards improvement in the intervention group was noted but our study lacked sufficient power to find a difference. In other settings validity and reliability in competency assessments with standardised patients has been shown to increase with the number of consultations examined. ^{31 32} Pragmatically, it was not feasible to measure multiple consultations in our study.

Errors in interrater measurement were minimised by using the same raters for all three periods of testing. The independent observer and patient were blind to study status but may have recognised the intervention group at the seven month follow up because of the learnt consultation styles. Other measures of competency were included to accommodate this unavoidable source of error.

Our study shows the potential of doctors to respond to the changing health needs of youth after brief training based on a needs analysis and best evidence based educational practice. Further study should address the extent to which these changes in doctors' competence translate to health gain for their young patients.

	Acknowledgments

We thank the participating doctors, Helen Cahill (Youth Research Centre, Melbourne University), Dr David Rosen (University of Michigan), and Sarah Croucher (Centre for Adolescent Health).

Contributors: LAS, the prinicipal investigator, initiated and conducted the intervention and wrote the paper. CC advised on recruitment, randomisation, pilot testing of instruments, and data analysis and helped write and edit the paper. FV provided advice, participated in pilot testing of instruments, provided the medical rater's assessment of doctors' skill, and helped edit the paper. MC-G helped to design and deliver the intervention and to edit the paper. GP advised on the study design and helped write and edit the paper. ND was a supervisor to LAS and advised on the evaluation methodology and helped edit the paper. GB was the chief supervisor to LAS and advised on the intervention design and evaluation and helped edit the paper. LAS will act as guarantor for the paper.

	Footnotes

Funding: The Royal Australian College of General Practitioners Trainee Scholarship and Research Fund and the National Health and Medical Research Council.

Competing interests: None declared.

	References

Top Abstract Introduction Participants and methods Results Discussion References

1.	Donovan C, Mellanby AR, Jacobson LD, Taylor B, Tripp JH. Teenagers' views on the general practice consultation and provision of contraception. The adolescent working group. Br J Gen Pract 1997; 47: 715-718[Medline].
2.	Oppong-Odiseng ACK, Heycock EC. Adolescent health servicesthrough their eyes. Arch Dis Child 1997; 77: 115-119[Abstract/Free Full Text].
3.	Ginsburg KR, Slap GB. Unique needs of the teen in the health care setting. Curr Opin Pediatr 1996; 8: 333-337[Medline].
4.	Veit FCM, Sanci LA, Young DYL, Bowes G. Adolescent health care: perspectives of Victorian general practitioners. Med J Aust 1995; 163: 16-18[Medline].
5.	McPherson A, Macfarlane A, Allen J. What do young people want from their GP? [Letter] Br J Gen Pract 1996; 46: 627[Medline].
6.	Bearinger LH, Gephart J. Interdisciplinary education in adolescent health. J Paediatr Child Health 1993; 29: 10-5S.
7.	Bennett DL. Adolescent health in Australia: an overview of needs and approaches to care. Sydney: Australian Medical Association, 1984.
8.	Veit FCM, Sanci LA, Coffey CMM, Young DYL, Bowes G. Barriers to effective primary health care for adolescents. Med J Aust 1996; 165: 131-133[Medline].
9.	Blum R. Physicians' assessment of deficiencies and desire for training in adolescent care. J Med Educ 1987; 62: 401-407[Medline].
10.	Blum RW, Bearinger LH. Knowledge and attitudes of health professionals toward adolescent health care. J Adolesc Health Care 1990; 11: 289-294[CrossRef][Medline].
11.	Resnick MD, Bearinger L, Blum R. Physician attitudes and approaches to the problems of youth. Pediatr Ann 1986; 15: 799-807[Medline].
12.	Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA 1995; 274: 700-705[Abstract/Free Full Text].
13.	Davis DA, Thomson MA, Oxman AD, Haynes RB. Evidence for the effectiveness of CME. JAMA 1992; 268: 1111-1117[Abstract/Free Full Text].
14.	Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. Can Med Assoc J 1995; 153: 1423-1431[Abstract].
15.	Owen JM. Program evaluation forms and approaches. St Leonards, NSW: Allen and Unwin, 1993.
16.	Davis D, Fox R. The physician as learner. Linking research to practice. Chicago, IL: American Medical Association, 1994.
17.	Greene JC, Caracelli VJ. Advances in mixed-method evaluation: the challenges and benefits of integrating diverse paradigms. New directions for evaluation, No 74. San Francisco: Jossey-Bass, 1997.
18.	Masters GN, McCurry D. Competency-based assessment in the professions. Canberra: Australian Government Publishing Service, 1990.
19.	Norman GR, Neufeld VR, Walsh A, Woodward CA, McConvey GA. Measuring physicians' performances by using simulated patients. J Med Educ 1985; 60: 925-934[Medline].
20.	Woodward CA, McConvey GA, Neufeld V, Norman GR, Walsh A. Measurement of physician performance by standardized patients. Refining techniques for undetected entry in physicians' offices. Med Care 1985; 23: 1019-1027[CrossRef][Medline].
21.	Rosen D. The adolescent interview project. In: Johnson J, ed. Adolescent medicine residency training resources. Elk Grove Village, IL: American Academy of Pediatrics, 1995:1-15.
22.	The Royal Australian College of General Practitioners College examination handbook for candidates 1996. South Melbourne: Royal Australian College of General Practitioners, 1996.
23.	Hays RB, van der Vleuten C, Fabb WE, Spike NA. Longitudinal reliability of the Royal Australian College of General Practitioners certification examination. Med Educ 1995; 29: 317-321[Medline].
24.	Bridges-Webb C, Britt H, Miles DA, Neary S, Charles J, Traynor V. Morbidity and treatment in general practice in Australia 1990-1991. Med J Aust 1992; 157: 1-57S.
25.	The general practices profile study. A national survey of Australian general practices. Clifton Hill, Victoria: Campbell Research and Consulting, 1997.
26.	Knowles M. The adult learner. A neglected species. Houston, TX: Gulf, 1990.
27.	Ward J. Continuing medical education. Part 2. Needs assessment in continuing medical education. Med J Aust 1988; 148: 77-80[Medline].
28.	Norman GR. Defining competence: a methodological review. In: Neufeld VR, Norman GR, eds. Assessing clinical competence. New York, NY: Springer, 1985:15-35.
29.	Rethans JJ, Strumans F, Drop R, van der Vleuten C, Hobus P. Does competence of general practitioners predict their performance? Comparison between examination setting and actual practice. BMJ 1991; 303: 1377-1380.
30.	Pieters HM, Touw-Otten FWWM, De Melker RA. Simulated patients in assessing consultation skills of trainees in general practice vocational training: a validity study. Med Educ 1994; 28: 226-233[Medline].
31.	Colliver JA, Swartz MH. Assessing clinical performance with standardized patients. JAMA 1997; 278: 790-791[CrossRef][Medline].
32.	Colliver JA. Validation of standardized-patient assessment: a meaning for clinical competence. Acad Med 1995; 70: 1062-1064[Medline].

(Accepted 7 October 1999)

Commentary: Applying the BMJ's guidelines on educational interventions

Jean Ker, lecturer in medical education. 

Clinical Skills Centre, University of Dundee, Ninewells Hospital and Medical School, Dundee DD1 9SY

jsker{at}dundee.ac.uk

In the western world, healthcare systems are facing enormous changes driven by both political and economic forces and by the increase in consumer expectations for competent and consistent quality health care. In response to these changes, medical education has become an increasingly important aspect of every doctors' professional life. Publishers have responded by including papers on medical educational issues with increasing frequency. This move has, however, required the development of guidelines to evaluate papers on educational interventions.

This critique applies guidelines developed by the BMJ's education group, which were published in the BMJ on 8 May 1999.

	Guideline 1: General overview

The commitment of the BMJ to publish more educational research makes the paper by Sanci et al an eminently suitable one for practising doctors interested in medical education.

Adolescent health care is challenging not only for general practitioners but for healthcare professionals involved in service delivery at all levels. This paper shows how successfully continuing medical education can be incorporated into changes in service delivery.

The principle steps of the educational intervention process are clearly outlined and can be generalised to other clinical settings, making it of interest to a wide readership. It contributes to the growing literature on evaluation of educational interventions in the general practice setting by attempting to show sustained changes in practice performance after a brief programme for continuing medical education.

The paper also follows the general style and guidelines for publication in the BMJ.

	Guideline 2: Theoretical considerations

One of the purposes of the guidelines on evaluating educational interventions is to facilitate, through papers, readers' understanding of the teaching and learning process so that they can apply any relevant aspects to their own practice.

In relation to this, the goals of this educational intervention are well described in the context of Australian general practice. The educational rationale was, however, rather brief in its explanation. An expanded discussion on the strategies used could have covered advantages and disadvantages. Readers may be able to utilise some of the learning opportunities given, but their links to the goals were not explicit.

	Guideline 3: Study presentation and design

A panel of stakeholders, including patients, was used to identify the content and design of the multifaceted intervention, which ensures the relevance of the intervention in terms of healthcare practice, and this was described in detail. The study design to ensure that standardised patients and observers were blind to the intervention status of the doctors is commendable.

In answering the questions posed in the guidelines some concerns with the design are raised.

The study is described as a randomised controlled study. A better and less misleading description would have been to describe it only as a randomised study, as it is often difficult to eliminate contaminants in an educational intervention. In fact the bias described in the type of practice, the language spoken, the age differences, as well as the college exams taken, does question the positive outcomes reported in the study.

The lack of a pretest to identify whether the two groups were comparable in terms of knowledge does also bring into question the final interpretation of the intervention. Purposive sampling based on a pretest and the variables described above would have been more appropriate and would have lent more meaning to the outcome.

The statistical analysis is clearly shared with the reader and well described. The use of a multifaceted evaluation system using recognised validated instruments reflects the guidelines for evaluating papers on educational interventions.

	Guideline 4: Discussion

The discussion was structured in accordance with the guidelines, with a clear statement of the principle findings. The sustainability of the intervention could, however, have been highlighted as it was a significant finding. The strengths and weaknesses of the study in relation to selection bias were well debated and justified.

The discussion in relation to other studies was, however, only briefly addressed, referring to only one systematic review of strategies for continuing medical education. This could have been expanded to support some of the findings, particularly in relation to the rapport and satisfaction of the standardised patients as a measurement of outcome.

The discussion did not begin to explore the implications for clinicians other than to indicate a need for assessing the health gain for patients from such interventions but did not discuss the difficulties of cost benefit analysis.

The guidelines on evaluating educational interventions as applied to this paper enabled the reviewer to systematically address all relevant aspects of the intervention. What is not clear is how much weighting should be placed on each guideline in relation to deciding whether the article should be published or not.

	Footnotes

website extra: The sample size calculation and a chart showing the flow of participants through the trial appears on the BMJ's website www.bmj.com