Learning in practice

Effect of ethnicity on performance in a final objective structured clinical examination: qualitative and quantitative study

Val Wass, senior lecturer in general practice ^a, Celia Roberts, senior research fellow ^b, Ron Hoogenboom, research assistant ^c, Roger Jones, Wolfson professor of general practice ^a, Cees Van der Vleuten, professor of education ^c. 

^a Department of General Practice and Primary Care, Guy's, King's, and St Thomas's School of Medicine, London SE11 6SP, ^b Department of Education and Professional Studies, King's College, London SE11 6SP, ^c Department of Educational Development and Research, University of Maastricht, Maastricht, Netherlands

Correspondence to: V Wass valerie.wass{at}kcl.ac.uk

	Abstract

Top Abstract Introduction Methods Results Discussion References

Objective: To assess the effect of ethnicity on student performance in stations assessing communication skills within an objective structured clinical examination.
Design: Quantitative and qualitative study.
Setting: A final UK clinical examination consisting of a two day objective structured clinical examination with 22 stations.
Participants: 82 students from ethnic minorities and 97 white students.
Main outcome measures: Mean scores for stations (quantitative) and observations made using discourse analysis on selected communication stations (qualitative).
Results: Mean performance of students from ethnic minorities was significantly lower than that of white students for stations assessing communication skills on days 1 (67.0% (SD 6.8%) and 72.3% (7.6%); P=0.001) and 2 (65.2% (6.6%) and 69.5% (6.3%); P=0.003). No examples of overt discrimination were found in 309 video recordings. Transcriptions showed subtle differences in communication styles in some students from ethnic minorities who performed poorly. Examiners' assumptions about what is good communication may have contributed to differences in grading.
Conclusions: There was no evidence of explicit discrimination between students from ethnic minorities and white students in the objective structured clinical examination. A small group of male students from ethnic minorities used particularly poorly rated communicative styles, and some subtle problems in assessing communication skills may have introduced bias. Tests need to reflect issues of diversity to ensure that students from ethnic minorities are not disadvantaged.

What is already known on this topic UK medical schools are concerned that students from ethnic minorities may perform less well than white students in examinations It is important to understand whether our examination system disadvantages them What this study adds Mean performance of students from ethnic minorities was significantly lower than that of white students in a final year objective structured clinical examination Two possible reasons for the difference were poor communicative performance of a small group of male students from ethnic minorities and examiners' use of a textbook patient centred notion of good communication Issues of diversity in test construction and implementation must be addressed to ensure that students from ethnic minorities are not disadvantaged

	Introduction

Top Abstract Introduction Methods Results Discussion References

Students from ethnic minorities seem to perform less well overall than white students in both undergraduate and postgraduate medical examinations.^1-4 Any form of potential racial discrimination within our examination systems is a cause for concern. ^{5 6} Problems with complex discourse may disadvantage students in oral examinations who have been trained overseas, but there is little further published work on the impact of differences in ethnicity on performance in examinations.⁷ This is becoming an increasingly important issue in undergraduate assessment. Fairness and consistency of assessment across UK medical schools is crucial.⁸ We need to understand any source of potential bias that may lead to racial disadvantage when developing tests for these skills.

When looking for potential discrimination within examinations, standardisation is a key issue; the more standardised the content, the less the potential for bias. Objective structured clinical examinations are currently most often used to assess undergraduate skills and include standardised simulated scenarios to test communication skills. ^{9 10} Yet it is still difficult to achieve true objectivity. ^{11 12} However carefully designed, the scenario presented to students will vary since neither simulated patient nor student is speaking from scripts. Examiners and simulated patients make judgments based on an impression of how well the student managed the consultation. This judgment, in turn, will be informed by their assumptions about what makes an effective consultation.¹³

We aimed to investigate whether students from ethnic minorities are disadvantaged by a bias in marking in a final year objective structured clinical examination, with a particular focus on stations assessing communication skills.

	Methods

Top Abstract Introduction Methods Results Discussion References

Our study took place in June 1999 during the final MBBS examination of the then Guy's and St Thomas's medical school. This comprised a three and a half hour objective structured clinical examination conducted over two days, consisting of two stations for history taking of long cases (21 minutes each) and 20 stations (seven minutes each) for clinical examination (nine stations), communication skills (six), and practical skills (five). The stations were similar but not identical on the two days. Simulated patients were professionally trained to standardise the scenarios used on communication stations.

A different examiner marked each station against a checklist and gave a final five point global rating for overall clinical competency. Simulated patients awarded a five point global rating for overall communication skills, independent of the examiner. The examiners and simulated patients had been briefed on the procedure. A minimum competence score for each station was set in advance, using the Angoff standard setting method.¹⁴

Each day we selected two communication scenarios, using role players from different ethnic backgrounds (table 1). The students gave informed verbal consent for video recording the scenarios. The local research ethics committee approved our study. Details of the students' ethnicity were made available after the examination.

The students were grouped as white, south Asian (Indian, Pakistani, Bangladeshi, Chinese, and Asian other), Afro-Caribbean, and other. For the purpose of our study, all students from ethnic minorities were categorised as one group, "ethnic minorities" (82 students) and all other candidates as "white majority" (97 students).

Quantitative analysis
We analysed the mean performance for stations for each day on all 22 stations, stations grouped by communication, practical, clinical skills, and long cases, and the specific study stations. We used an independent two sample t test to examine relations between student performance and ethnicity. We regarded P values greater than 0.01 as non-significant. The reliability of each objective structured clinical examination was calculated with Cronbach's .

Qualitative analysis
All video recorded encounters were viewed as well as recorded comments made by simulated patients and examiners after students had left the station. The duration of the interaction, student's ethnicity, and observations made during the viewing were recorded on a standard form. An assessment was made of the extent to which simulated patients and students established a relatively patient centred encounter. Any potential misunderstandings, false assumptions, or explicit discriminatory behaviour were noted. The analyst (CR) viewed the encounters "blind," allocating grades and comparing these after the viewing with the ratings from the simulated patients and examiners. Discrepancies between analyst, simulated patient, and examiner were recorded.

We used these records to select specific interactions for detailed transcription. We identified recurring themes, which acted as background to the detailed discourse analysis. We used these to clarify the complexity of the doctor-patient consultation and the communicative demands it placed on students.¹⁵

	Results

Top Abstract Introduction Methods Results Discussion References

We excluded four of the 179 students, as their ethnicity was undeclared. Table 2 gives the ethnicity, sex, and age of the remaining 175 students. Seventy eight (45%) were from ethnic minorities. All but two students had received secondary school education in the United Kingdom.

Table 3 shows the mean performance of the students in the overall examination and in the specific study stations. Mean performance on communication stations was significantly higher for white majority students than for students from ethnic minorities on both day 1 (72.3% (SD 7.6%) 67.0% (6.8%); P=0.001) and day 2 (69.5% (6.3%) and 65.2% (6.6%); P=0.003). The Cronbach  reliability of the objective structured clinical examination was 0.74 and 0.76 on days 1 and 2, respectively.

We were unable to assess 49 (14%) of the video recorded interactions due to technical faults. We found no explicit examples of breakdown in communication or of discriminatory behaviour in the remaining 309 interactions. Neither simulated patients nor students showed, through talk or bodily movements, any expression identifiable as a negative response to the other's ethnicity.

For detailed discourse analysis we transcribed 28 (9%) interactions, representing a range of scores from high to low and including students from both groups. Two main findings emerged.

Firstly, students created different interactional climates. Those receiving high grades were relatively empathetic, responsive, and persuasive, building a joint problem solving framework with the patient. Conversely, some failed to build this framework, displayed various moves to distance themselves from patients, and were given low grades by both examiners and simulated patients.¹⁶ Students from both groups failed to create this interactive framework (see box and bmj.com). Relatively more male students from the ethnic minority group were in this category: Fifteen (12 male) of the 22 students scoring below minimum competence were in the ethnic minority group. In these instances there were no obvious cultural and linguistic differences, although these students were more likely to have pronunciation, word stress, and intonation influenced by their heritage language.

Framework for good and poor communicative styles The framework for communicative style consisted of four levels: Performance factorsthese included clarity, slips of the tongue, hesitations, voice quality, and aspects of non-verbal communication The design of questions and responsesfor example, the ways in which students showed that the patient's problem needed to be jointly managed, or the ways in which students were sensitive to the needs of the patient; or by contrast the negative labelling of the student or the use of "trained" empathy The overall thematic staging of the consultationfor example, a student who had to resist giving a methadone prescription to a drug addict managed the consultation so as not to either give in or refuse too early on; or, by contrast, a student shifting rapidly from one topic to another and preventing the simulated patient from following the student's line of reasoning Ideological positioning of the studentfor example, how much to rely on personal authority and how much on medical authority

Secondly, there were instances where the examiner gave top marks but the simulated patient from an ethnic minority gave a lower mark. These students tended to use a style in which explicit guidance was deferred, there was more talk about the nature of the consultation, and there was more talk about talkfor example, "But first I'd like to know a little more about you" (see bmj.com). Although this style fitted well with the white examiners' textbook notions of a patient centred consultation, it was not rated so highly by some simulated patients from ethnic minorities.

	Discussion

Top Abstract Introduction Methods Results Discussion References

We found differences in the mean performance of students from ethnic minorities on communication stations in the final year objective structured clinical examination. Although we saw no obvious evidence of breakdowns in communication or of discriminatory judgments, two subtle differences emerged: a particularly poor communicative style that may have distanced some students from ethnic minorities from the simulated patient¹⁶ and instances where the examiner's assumptions did not match the expectations of the simulated patients.

We do not want to claim too much from these two points: it would be wrong to reify ethnicity and assume problems and differences in communicative style on the basis of an individual's ethnicity alone. Similarly, our comments on the differences between examiners and the simulated patients from ethnic minorities are speculative and need further investigation. In combination, however, these two factors may account for at least some of the differences in the ratings of white students and students from ethnic minorities.

The style of some students to distance themselves from patients reflects a medical model of consultation rather than a more social one preferred by examiners. Students from ethnic minorities might be more likely than white students to use this style because a medical model is less demanding of communication skills and may be perceived as appropriate. In this case, differences in motivation and learning styles also need to be considered.¹⁷ So several complex factors, styles of communication, values, and ways of learning may all be important and may be related to the ways in which students are socialised into medical school culture.

Students who live outside the medical school and have networks of family and friends where there are quite different communicative experiences from those in the university may be less exposed to the informal and social talk around medicine occurring in institutional life. ^{18 19} They may therefore have less opportunity to tune into the current institutional norms about what counts as a good consultation.

There is a case for developing a wider repertoire of communicative styles for setting the stations. Important questions have been raised for educators in cross cultural communication in medicine, but this needs to be addressed at the local levelfor example, by examiners working with a range of non-traditional students and simulated patients from ethnic minorities to build up a wider repertoire of styles sensitive to the diverse backgrounds of patients.¹³ Institutions may also not be aware of hidden processes that reward some students and penalise others in final examinations.

	Acknowledgments

We thank Professor Gwyn Williams and Dr Charles Twort for approving this study, Stevo Durbaba for his technical assistance, Phil Doulton of Professional Role Play, Nora Edmead for her administrative support, and the students and examiners for their cooperation.

	Footnotes

Funding: The King's Fund.

Competing interests: None declared.

Extracts from transcripts appear on bmj.com

	References

Top Abstract Introduction Methods Results Discussion References

1.	McManus IC, Richards P, Winder BC, Sproston KA. Final examination performance of medical students from different ethnic minorities. Med Educ 1996; 30: 195-200[Web of Science][Medline].
2.	Dillner L. Manchester tackles failure rate of Asian students. BMJ 1995; 310: 209[Free Full Text].
3.	Wakeford R, Farooqi A, Rashid A, Southgate L. Does the MRCGP examination discriminate against Asian doctors? BMJ 1992; 305: 92-94[Abstract/Free Full Text].
4.	Jolly BC, Cohen R, Rothman AI, Ross J. Graduates of foreign medical schools: demographic and personal predictors of success on an OSCE-format internship programme entrance examination. In: Proceed-ings of the 27th annual conference of the Association of American Medical Colleges. Washington, DC: Association of American Medical Colleges, 1988:234-239.
5.	Smith R. Prejudice against doctors and students from ethnic minorities. [Editorial.] BMJ 1987; 294: 328-329[Free Full Text].
6.	McKenzie KJ. Racial discrimination in medicine. [Editorial.] BMJ 1995; 310: 478-479[Free Full Text].
7.	Roberts C, Sarangi S, Southgate L, Wakeford R, Wass V. Oral examinationsequal opportunities, ethnicity, and fairness in the MRCGP. BMJ 2000; 320: 370-374[Free Full Text].
8.	Catto G. Education, education, education. BMJ Classified 23 Jun, 2001:2-3.
9.	Fowell SL, Maudsley G, Maguire P, Leinster SJ, Bligh J. Student assessment in undergraduate medical education in the United Kingdom. Med Educ 2000; 34(suppl 1): 1-49[Web of Science][Medline].
10.	Wass V, Vleuten C van der, Shatzer J, Jones R. Assessment of clinical competence. Lancet 2001; 357: 945-949[CrossRef][Web of Science][Medline].
11.	Vleuten CPM van der, Swanson DB. Assessment of clinical skills with standardised patients: state of the art. Teach Learn Med 1990; 2: 58-76.
12.	Colliver JA, Verhulst SJ, Williams RG, Norcini JJ. Reliability of performance on standardised patient cases: a comparison of consistency measures based on generalizability theory. Teach Learn Med 1989; 1: 31-37[Medline].
13.	Skelton JR, Kai J, Loudon RF. Cross-cultural communication in medicine: questions for educators. Med Educ 2001; 35: 257-261[CrossRef][Web of Science][Medline].
14.	Angoff WH. Scales, norms, and equivalent scores. In: Thorndike RL, ed. Educational measurement, 2 ed. Washington, DC: American Council on Education.
15.	Erickson F, Shultz J. The counsellor as gatekeeper: social interaction in interviews. London: Academic Press, 1982.
16.	Roberts C, Wass V, Jones R, Sarangi S, Gillett A. A discourse analysis study of "good" and "poor" communication in an OSCE: a proposed new framework for teaching students. Med Educ 2003 (in press).
17.	McManus IC, Richards P, Winder BC, Sproston KA. Clinical experience, performance in final examinations, and learning style in medical students: prospective study. BMJ 1998; 316: 345-350[Abstract/Free Full Text].
18.	Gumperz J. A discussion with John Gumperz. In: Eerdmans S, Previgagno C, Thibault P, eds. Discussing communication analysis 1 John Gumperz. Lausanne: Beta Press, 1997.
19.	Atkinson P. Medical talk and medical work: the liturgy of the clinic. London: Sage, 1995.

(Accepted 20 November 2002)