How appropriate is the situational judgment test in assessing future foundation doctors?BMJ 2023; 380 doi: https://doi.org/10.1136/bmj.p101 (Published 13 January 2023) Cite this as: BMJ 2023;380:p101
Each December and January more than 9000 final year medical students sit the situational judgment test (SJT),1 which forms a large part of their applications for UK foundation doctor posts. Despite the exam being national and compulsory, students sit the test externally to their medical schools at national centres, with a booking system that rivals that of securing Glastonbury tickets.2
The purpose of the exam is to assess some of the “essential competencies” of foundation doctors, such as “patient focus, commitment to professionalism, coping with pressure, effective communication, and team working.”3 It is often referred to as “a test of employability.”1 The exam outlines a series of scenarios in a clinical workplace and asks multiple choice questions about how a foundation doctor should respond. Candidates must either choose the best three answers, rate the appropriateness of all the responses, or rank every response from best through to worst.
The exam is always met with a great deal of frustration from students, as the amount of preparation you can do is limited.4 Compared with the many other exams that students sit during medical school, the SJT does not test memory recall or clinical knowledge and has fewer resources and guidelines available. The exam also makes up 50% of each student’s national ranking, which determines where in the UK students will be placed for the next two years. The other half of the points are based on students’ achievements throughout all their years at medical school. If a student performs poorly in this one exam, they might have to move across the country, risking isolation from friends, family, or social support networks.
As a current final year student, I was one of the 9905 students who sat this exam this season.1 As I was revising for the test, I began to feel increasingly frustrated by the questions. Examples of the scenarios and marking rationale that have been used in previous years regularly crop up in Twitter discussions among medical students and doctors,56 highlighting numerous problems with both the content and structure of the test.
Frustrating, inconsistent, and biased
Many of the questions are designed to rank a set of responses from most to least appropriate for a given scenario. Yet I often found that there were no options that felt representative of what you might realistically do. In most cases, there are options that a doctor would perhaps never do, yet candidates are still required to rank which of these isolated statements are worse than the other responses. If these actions would not be taken anyway, does it matter in which order candidates rank them? The style of the questions does not allow for any nuance or complexity nor any of the human factors that typically affect our day-to-day interactions.
The tests are also a source of differential attainment, with studies showing that white candidates score significantly higher than ethnic minority candidates on the SJT.7 It is difficult to ascertain why this might be—perhaps the rigidity of the questions enforce cultural biases in our society? When speaking at a webinar organised by the BMA on the process, Amir Sam, head of Imperial College School of Medicine, questioned why we are using assessment tools that seem to discriminate against candidates based on ethnicity, rather than grading them by aptitude.1
The rationale for the marking scheme is also flawed and often differs between questions. In one question you are expected to apologise for a colleague, yet in another you should not—despite the two scenarios being extremely similar. The marking scheme and chosen “correct” answers are decided by a “panel of experts.” Yet research has found that only two thirds of these experts agree on what the best answer is, and they often find it “difficult to commit themselves to a single best answer as instructed” or note that none of “the proposed answer options [are] adequate.”8 The available answers never include an option that allows the theoretical doctor to stand up for themself or their rights in the system. A common thread running through each question is that every situation is an individual person’s responsibility—the test fails to account for how larger, systemic factors will often shape how a doctor can and should respond.
For such a high stakes exam, it is alarming that it is beset by so much inconsistency and bias. Why are we setting questions for students that even experts cannot agree on? If the exam is designed to assess candidates’ non-academic attributes, why is the format so rigid? Does asking students to accept facts at face value and not ask any follow-up questions or seek clarifications prevent our future doctors from thinking critically? For every question I read when revising for the SJT, I could have come up with a further five follow-up questions that would inform how I would respond in practice. Surely “key attributes” for doctors should include curiosity and asking questions—the very basis of what we are taught in medical school?
Discussions between the BMA and the UK Foundation Programme Office about changing the foundation allocation process are taking place for as early as the 2024 application cycle.1 Given the frustrations that students and other stakeholders have long voiced about the SJT, we need to consider whether this is the right format for assessing our future foundation doctors.
Competing interests: NN is a medical student enrolled at the University of Manchester 2017-23.
Provenance and peer review: not commissioned; not externally peer reviewed.