Predicting the “strugglers”: a case-control study of students at Nottingham University Medical SchoolBMJ 2006; 332 doi: http://dx.doi.org/10.1136/bmj.38730.678310.63 (Published 27 April 2006) Cite this as: BMJ 2006;332:1009
- Janet Yates, research fellow,
- David James, director of medical education ()
- Medical Education Unit, Faculty of Medicine and Health Sciences, University of Nottingham, Queen's Medical Centre, Nottingham NG7 2UH
- Correspondence to: D James
- Accepted 17 November 2005
Objective To identify potential predictors of undergraduate students who struggle during their medical training.
Design Case-control study. Cases were students who had experienced academic or personal difficulties that affected their progression on the course (“strugglers”). Controls were selected at random from the corresponding year cohorts, using a ratio of four controls for each struggler.
Setting University of Nottingham Medical School.
Participants Students who entered the course over five consecutive years.
Main outcome measures Likelihood ratios for independent risk factors for struggling on the course
Results 10-15% of each year's student intake were identified as strugglers. Significant independent predictors of students being in this category were negative comments in the academic reference (likelihood ratio 2.25, 95% confidence intervals 1.44 to 3.50), lower mean examination grade at A level (2.19, 1.37 to 3.51), and the late offer of a place (1.98, 1.19 to 3.30). Male sex was a less significant risk factor (1.70, 1.09 to 2.65) as was a lower grade at GCSE science (2.13, 1.12 to 4.05). In UK students whose ethnicity was known, not being white was a significant predictor of struggling (2.77, 1.52 to 5.05) but the presence of negative comments was not. Age at entry to the course and the possession of a previous degree were not predictive.
Conclusions Our results support retention of existing selection practices relating to academic achievement and critical review of students' references. We plan to undertake further investigation of the reasons why some students, including men, those with late offers and those from ethnic minority backgrounds, may do less well on the Nottingham course.
Selecting the “right” students is a challenge for medical schools and the subject of much debate.1 2 3 Most medical schools no longer select solely on the basis of high academic qualifications but include varied non-academic criteria. The aim is to identify personal qualities in potential students that will allow them to cope with the rigours of the medical course and to become globally competent as practising doctors.4 The General Medical Council (GMC) has emphasised that curricular objectives should cover not only knowledge but also skills, attitudes, and behaviour.5 At Nottingham, there is regular review and development of the admissions process, which currently comprises four stages: review of academic ability, scoring of a validated questionnaire that focuses on personal attributes and attitudes, review of the statements on the application form from UCAS (the Universities and Colleges Admissions Service), and a semistructured interview by two trained interviewers.
It is not surprising that some students have doubts about their chosen career or have difficulties coping with the course, given the age of most medical students when they are selected for medical school, the limitations of current selection methods, the individual development that can occur over the five years of a course, and the different approaches to learning and assessment that students encounter at university. Yet to date, most studies of student performance have focused on positive predictors of successful students (such as A level grades (UK state examinations generally taken at age 17-19), personality, and learning styles).1 While important, such studies do not provide information about factors that predict impaired performance or students who may struggle both academically and personally. Indeed factors that predict success may be qualitatively different from those that predict problems. Identifying these would not only fill a gap in the current literature but also inform selection and support processes.
A medical student's training represents a major investment by the government. Currently, for the five year undergraduate course at Nottingham, this includes over £50 000 ($87 400, €73 580) per student from the Higher Education Funding Council for England (HEFCE) plus about £125 000 ($218 534, €183 994) per student for clinical placements via the Service Increment for Teaching (SIFT). The period at medical school also represents a considerable personal investment of time and money for the student. For both the good of the individual and of society then, it is important to minimise the rate of attrition by optimising the student selection procedure and having robust mechanisms to identify and support those who are “struggling.”
We conducted a case-control study of struggling students over five successive cohorts of medical student entrants at Nottingham to determine whether any preadmission factors, academic or non-academic, predicted an increased risk of problems on the course.
Identification of strugglers and controls
We studied five consecutive year groups of medical students who should have completed the whole course at the time of data collection. The dates are not specified precisely to protect the confidentiality of the individuals in the study. We identified struggling students in one of four ways: students who attended the academic progress committee (APC), which meets several times each year and interviews students experiencing major academic difficulties and agrees various strategies of support; students who had their course terminated, usually for academic reasons; students who left the course voluntarily, whether for personal or academic reasons; and students whose course was suspended temporarily, for personal or academic reasons. These categories were not necessarily mutually exclusive—for example, a student who attended the academic progress committee might subsequently leave the course.
For each struggler, we identified four control students from the same year group cohort, but who did not fulfil the above criteria, by a computer generated randomisation programme that used each student's unique course identification code. We chose the 1:4 case:control ratio after statistical advice to minimise bias through chance yet avoid excessive data entry.
For all students (strugglers and controls) we extracted data manually from two sources: the UCAS forms submitted by the students at the time of selection and the students' progress records once they were on the medical course. We supplemented these data by additional information provided directly by UCAS. Table 1 gives the details.
GCSE (general certificate of secondary education) and A level passes were scored (at GCSE, A* = 6, A = 5 etc, and at A level, A = 10, B = 8, etc) to create mean examination grades (MEGs). We used mean grade cut-off thresholds of < 5 for GCSE (5 is the equivalent of an average of grade A for all GCSE subjects, the current GCSE screening threshold in our selection process) and < 9.33 for A level (9.33 is the equivalent of AAB grades; the minimum required A level grades during the period of the study were ABB but most students exceeded this level).
“Negative comments” were remarks made in the academic references on the UCAS form and generally referred to possibly adverse aspects of personality or behaviour. They were extracted verbatim and then summarised as a yes/no variable (details are available from the corresponding author on request). In general the negative comments fell into one of five categories: academic (such as lack of ability, motivation, or self discipline); personality (such as excessive anxiety, lack of self confidence, or reluctance to participate); attitude (such as intolerance of others, arrogance, overconfidence); ongoing medical or social problems (such as chronic debilitating illness, severe family problems, or poor adaptation to cultural change); and late decision to read medicine (lack of appropriate work experience). We validated our assessment of what constituted a negative comment by asking 23 (50%) of the members of the current interviewing panel, who were not aware of the details of the study, to assess 25 comments as potentially positive, negative, or neither, by means of a one page postal questionnaire. We included eight comments that we thought were “positive” among 17 negative ones. We assessed inter-rater reliability of the identification of negative comments by asking an independent researcher to review 62 statements (a 10% random sample) against our criteria of negative comments.
We used SPSS v11 for data analysis. Inter-rater reliability of the identification of negative comments was tested with the κ statistic. Our primary outcome variable was being a struggler. For univariate analysis we compared dichotomous variables with χ2 tests and continuous variables by Mann-Whitney tests. For multivariate analysis we tested for independent predictors by binary logistic regression, using the backwards logistic regression option and including significant univariate factors in the modelling. We included year of entry to the course as a categorical variable to adjust for change over time.
Overall, 123 students from the five cohorts were identified as strugglers and there were 492 controls. Fifty three strugglers were initially identified as APC attendees, nine as course terminations, 30 as voluntary withdrawals, and 31 as course suspensions. Thus of the 961 students admitted over the five years, 123 (12.8%) experienced problems, including 41 (4%) who left the course before graduation, as described below.
Extent of strugglers' problems
The strugglers experienced considerable academic problems. Seventy of the 123 (57%) failed three or more preclinical exams and only 92 (75%) had graduated as BMedSci (bachelor of medical science) at the end of the third year; of the 31 remaining, 21 left the course voluntarily, eight had their course terminated, and two were still on the preclinical course. At a date when all students should have completed their training, a further 12 had left during the clinical course, eight were still on the clinical course, and only 72 (59%) of the original group of strugglers had graduated BMBS (bachelor of medicine and bachelor of surgery).
In addition, 56 of the 123 (46%) had disruptive personal or medical problems noted in their undergraduate records, including a high incidence of depressive illness (strugglers 29/123 (24%) v controls 6/492 (1%); χ2 91.6, P < 0.001). Overall, 34 (28%) of the strugglers had attitude problems noted in their undergraduate records.
Preadmission factors associated with strugglers (univariate analyses).
Table 2 and Table 3 summarise these results. The non-academic factors that were significantly more common in the strugglers were the presence of negative comments in the head teacher's reference, the late offer of a place, male sex, and slightly older age. Non-white ethnicity was significant in UK students. There was no significant difference between UK strugglers and controls in social deprivation (Townsend score6).
The significant academic associations with the strugglers were lower mean examination grade at GCSE, lower mean examination grade at A level, and not achieving a grade A at A level biology. Such students were also more likely to have significantly lower UCAS points (point score for top three A levels) and total tariff scores (point score for all A levels taken). There were no significant differences between the groups in terms of A level chemistry grade or having undertaken a previous degree.
Preadmission factors significantly and independently predicting strugglers (multivariate analysis)
We entered preadmission factors that had shown significant univariate associations with struggler status (above) into a binary logistic regression. Table 4 shows the data for all students excluding ethnicity as an explanatory variable because it was not known for overseas students, and data for UK students including ethnicity as that was recorded for most of these.
A lower mean examination grade at A level and the late offer of a place were highly significant predictors in all students (likelihood ratio 2.19, 95% confidence interval 1.37 to 3.51, and 1.98, 1.19 to 3.30, respectively) and in UK students alone (2.36, 1.40 to3.99, and 2.25, 1.27 to 3.99). Negative comments were strongly predictive in the whole group (2.25, 1.44 to 3.50) but not in UK students alone. Men and those with lower GCSE science scores were more likely to struggle in the entire group (1.70, 1.09 to2.65, and 2.13, 1.12 to 4.05) but again not in UK students alone. Non-white ethnicity was a highly significant predictor in UK students (2.77, 1.52 to 5.05).
Validation of negative comments
Eighteen members of the interviewers' panel returned completed questionnaires (78% response rate). For the 17 comments that we (JY and DJ) thought were potentially negative, overall the interviewers agreed they were more likely to be negative. For the eight comments that we (JY and DJ) considered positive, again, the interviewers agreed they were more likely to be positive. The results of the negative comment survey are available from the corresponding author.
The review of 62 statements by an independent researcher revealed satisfactory inter-rater agreement with the identification of negative comments (κ = 0.69).
Our study shows that students with lower A level grades had the greatest risk of experiencing problems during medical training in the five consecutive cohorts studied. The late offer of a place was also associated with risk and men were slightly more at risk than women. When we excluded ethnicity as a pre-admission variable, negative comments were strongly predictive, but in UK students non-white ethnicity was a significant risk factor.
Strengths and limitations
We collected a wide range of robust quantitative data, with a 1:4 case:control ratio and random selection of controls to avoid matching on discriminatory factors. We tested the implicit assumption that our controls were representative of the year groups by comparing them with non-selected students—that is, those who were neither strugglers nor controls—for five pre-admission factors (age, sex, where they normally live, A level points score for the top three grades, and having a previous degree). There were no significant differences. We were unable to test the impact of ethnicity for the whole group as data were unavailable for overseas students.
In comparing strugglers with controls we chose to analyse our strugglers as a single group because there was no definitive way to split them up. Their problems comprised academic, personal, medical, or social components in varying degrees. Hence, we could define a struggler only as a student who failed to make satisfactory progress and was identified in any one of the categories listed. We accept that there may be subtle associations between the categories and the predictors that we were unable to identify, but a subgroup analysis of those strugglers who attended the academic progress committee and those whose course was terminated showed that the same range of factors emerged as predictors of struggling (see bmj.com for results).
The inclusion of “negative comments” as a defining variable can be criticised on the grounds of subjectivity, especially as there was a wide range of comments and the researcher (JY) was not blind to control/struggler status. The validation that was carried out in the form of a questionnaire distributed to interviewers and the independent review of statements provided some justification for the scoring. However, in an unstructured reference in which teachers are usually trying to accentuate their pupils' positive characteristics, we accept that the scope for finding negative ones may be limited. Furthermore, the scoring is still subjective. Additional studies using more formal statement analysis techniques, which we have used in previous research, are under way.7
Although we could not fully assess the independent predictive value of ethnicity in the whole group, we would expect non-white status to be comparably predictive in non-UK students as they are predominantly of non-European origin and also have a higher proportion of strugglers (22/86 (26%) v 101/529 (19%) of home students).
Comparison with other studies
We chose a different design and outcome to many other studies, which have used full cohorts and measured rates of examination success. However, our work confirms that risk factors for failure—lower school examination grades, non-white ethnicity, and being male—were as expected, being the opposite of those factors that have been reported to predict success.1 8 9 10 11
We are not aware of any other studies that have used “negative comments” in the academic references. Powis et al found that interviewers' negative remarks had some predictive value for course withdrawal in a study using 1:1 partial case-control matching, but the association was relatively weak.12 Papadakis et al suggested some correlation between negative statements made during the undergraduate years regarding professional conduct and later disciplinary action in graduates.13 A previous study at Nottingham showed no relation between the number of positive themes in the academic reference and course success, but the reverse was not examined.7 We found many negative comments that conveyed the opposite of those factors that are linked to success,7 14 15 although they tended to be less clear cut. Statement review is an integral part of the selection process at Nottingham, and our results confirm that interviewers should be alert for negative comments, not in a discriminatory sense but to explore them more fully with the student.
Similarly, we are not aware that others have examined the timing of course offers in relation to undergraduate progress, but our study shows that those who fail to impress the interviewers initially or apply at a late stage, perhaps after rejection by other schools or through “clearing,” are at greater risk of struggling than those who received an early offer of a place.
Future policy and research
Many medical schools in the UK are exploring more varied admissions policies, perhaps incorporating elements of the successful Australian policies of lower examination grades accompanied by psychometric testing.3 Their outcome evaluations, especially in relation to non-traditional students, may be important in guiding future policy across the UK.2 16 17 Our data suggest that the current four stage approach to student selection is sound, but we now have concerns that the introduction by UCAS of open references will reduce the opportunities for head teachers to draw attention to personal qualities or difficulties that might make it difficult for a student to succeed in medicine; structured references might be more helpful.
Pastoral support at Nottingham includes informal meetings with personal tutors and more intensive formal mechanisms, yet some students still hide, or deny, their difficulties until they reach a crisis point. In the course of this research we noticed a high incidence of depressive illnesses in strugglers, which is of particular concern. We intend to review our strugglers and our pastoral practices more closely to see what further support could be offered, perhaps as targeted interventions to those at greatest risk. Research elsewhere has identified personal, social, cultural, and financial pressures that may particularly affect students from non-mainstream backgrounds and that may need to be addressed explicitly and proactively.18 19 Failure in clinical examinations may have a sex related or cultural basis because the current emphasis on patient centred, empathetic care may be more natural for women than for men20 and may present a considerable difficulty for students from more paternalistic cultures.21 22 23 24 Language barriers may be important because fluency in standard English may not be adequate for medical and colloquial needs.25 26 27
In conclusion, this study supports our existing selection practices at Nottingham relating to previous academic achievement and the critical review of students with negative references. Those who were initially rejected or who were “reserves” may require additional support during their course. We plan further investigations into the nature of the negative comments and the characteristics and difficulties of those who do less well or fail on the course.
What is already known on this topic
Male students and those with lower A level grades and of non-white ethnicity may perform less well on the undergraduate medical course
What this study adds
Late acceptance on to the course and the presence of “negative comments” in the academic reference are additional risk factors at Nottingham medical school
A subanalysis of strugglers is on bmj.com.
The 1991 Townsend deprivation scores were accessed via the Census Data Service at http://census.ac.uk/cdu/Datasets/1991_Census_datasets/. We are grateful to Chris Rix of UCAS for providing approval and data for Nottingham applicants specifically for this study and to Carol Coupland for statistical advice. We thank David Powis and Eamonn Ferguson for reviewing an early draft of this paper and for providing valuable suggestions and advice, Caroline Mulvaney for performing statement validation, and members of the interviewing pool who responded to our questionnaire on negative comments.
Contributors DJ conceived the study and is guarantor. JY collected and analysed the data. Both authors contributed to interpretation and wrote the paper.
Funding JY is paid by Service Increment for Teaching (SIFT).
Competing interests None declared.
Ethical approval Faculty of Medicine research and ethics committee.