Getting into the foundation programme: the new selection methodsBMJ 2012; 345 doi: https://doi.org/10.1136/bmj.e6099 (Published 25 September 2012) Cite this as: BMJ 2012;345:e6099
- Siobhan Fitzpatrick, project manager, Improving Selection to the Foundation Programme, and senior policy officer1,
- Paul O’Neill, chair, Improving Selection to the Foundation Programme project group1, professor of medical education2, honorary consultant physician3
- 1Medical Schools Council, London, UK
- 2University of Manchester, UK
- 3University Hospital of South Manchester NHS Trust, UK
Siobhan Fitzpatrick and Paul O’Neill outline the changes to selection into the foundation programme for 2013
Selection of applicants to the foundation programme has changed considerably since a national system was introduced in 2005. The introduction of a situational judgment test (SJT) and educational performance measure (EPM) for applicants to foundation training from August 2013 represents a further step in the evolution of selection. The implementation of the SJT and EPM is based on research, development, extensive piloting, and wide ranging consultation with stakeholders to ensure that new doctors are selected in the most valid, reliable, and effective way.
The best practice in selection uses several approaches to ensure that applicants’ cognitive and non-cognitive skills are taken into account. In the new system for the foundation programme, an SJT replaces “white space” questions (free text questions that are mapped against the national person specification) as a measure of likely professional judgment and behaviour; and the EPM replaces “academic quartiles” as a measure of clinical and non-clinical knowledge and skills. Although the selection tools have changed, the overall application process remains largely the same, including eligibility requirements, local allocation to posts, and additional processes for the academic foundation programme and the Defence Deanery.
What are the SJT and the EPM?
An SJT is a measurement method designed to assess a range of attributes and professional behaviours required for a job. Such tests are increasingly used in large scale selection processes and have been used for decades by the US Federal Bureau of Investigation, the UK National Policing Agency, and more recently for selection to general practitioner training.
Research has consistently shown that, when designed appropriately, SJTs show good reliability and validity as a predictor of future job performance.12 Given this, it is essential that the test is closely based on the work that a foundation doctor does. As part of the Medical Schools Council’s Improving Selection to the Foundation Programme (ISFP) project, a detailed job analysis3 was completed that involved observing and interviewing foundation doctors and interviewing supervisors and other key people. The analysis identified overlapping attributes that could be organised into five groups:
Commitment to professionalism
Coping with pressure
Patient focus, and
Working effectively as part of a team.
The SJT for selection to the foundation programme will consist of quality assured, multiple choice items that are based on these domains and linked to the person specification for a foundation year 1 doctor.4 These items are written by clinicians working closely with or as foundation doctors, and each item is based on a scenario typically encountered by first year doctors. Each item requires the applicant to indicate what they should do in that scenario, either by choosing the three best of eight responses or ranking five responses in decreasing order. The marking key is agreed in discussion with foundation doctors and panels of senior clinicians.
The EPM is a composite of performance assessments in clinical and non-clinical skills and knowledge at medical school up to the point of application to the foundation programme. It is an evolution of the previous academic quartiles, in that UK medical schools have agreed a set of standardised principles for use of assessment data across a range of years and formats, which gives schools flexibility in determining the most appropriate measures locally. The EPM is based on a “basket of assessments” that reflects the curriculum and as such is a measure of performance throughout medical school. Consequently, it will recognise consistent work and achievement, with likely weighting towards the later years.
Each school has consulted with its students to ensure that the development of the EPM is transparent and based on that school’s assessments. Students at the start of their final year will be ranked into decile groups. Other academic achievements are recognised with up to five points for additional degrees and two points for national or international prizes, presentations, or publications.
The rationale for change
The selection system used until this year—which involved ranking applicants by using a combined score derived from answers to a set of white space questions and an academic quartile ranking provided by the applicant’s medical school—has been the source of some concern (box 1). The 2008 Next Stage Review of the NHS recommended that better tools be developed for selection into the foundation programme.5
The Medical Schools Council was asked to lead a stakeholder project group—with representation from medical students, employers, foundation schools, and the General Medical Council—to appraise the options for improving selection and pilot the recommended options. The 2009 option appraisal involved a detailed evaluation of selection methods such as interviews, assessment centres, and a national exam.6 Three literature reviews of international best practice were commissioned, and extensive consultation with stakeholders informed the non-financial considerations that fed into the cost-benefit analysis.
The preferred options—the SJT and EPM—were then developed specifically for use in the context of foundation programme selection (box 1). Between 2010 and 2012, more than 8000 final year students and every UK medical school participated in and gave feedback on pilots. More than 150 clinicians have been trained in SJT item writing; a standardised EPM framework has been agreed; and specific standards for the administration and secure delivery of the SJT and EPM have been rigorously tested. Participants in the pilots have responded positively, reporting that the SJT was fair and relevant, and detailed analysis has confirmed the psychometric quality of the test.7 Ongoing evaluation is planned.
Box 1: Rationale for change
Concerns about previous system
The limited range of new “white space” questions meant concerns about sustainability
Marking of white space questions was labour intensive (clinician time)
Use of white space questions in non-invigilated conditions, and availability of model answers on the internet, raised concerns about plagiarism and coaching
Technical reliability and validity were open to question
The academic quartile system made it difficult to standardise the process across medical schools
Improvements through the SJT and EPM
The situations experienced in the foundation programme are varied and complex, so new items can be written continuously to build a large bank of items
Some clinicians are involved in item development, but each item is used for several thousand applicants. SJTs are machine markable
The SJT is taken in invigilated conditions in the UK. It is not possible to revise for the SJT because the scenarios are complex and answers relate to judgment rather than knowledge
Evidence for the validity of SJTs has accumulated over the years. Analysis of the large scale SJT pilots demonstrated their reliability and validity
The EPM framework achieves transparency and consistency while maintaining local flexibility. The measure requires schools to adhere to a standard framework to calculate rank in relation to cohort. The move to deciles is fairer to applicants at the borders
How the SJT and EPM will be used in the application process
As before, applications for entry to the foundation programme are submitted through the online Foundation Programme Application System, which is managed by the UK Foundation Programme Office. The SJT and EPM are each worth 50 points of the 100 point application score that is used, alongside applicants’ preferences, to select applicants to individual foundation schools. These schools then have their own processes to allocate applicants to specific programmes.
As described above, the EPM comprises two elements: medical school performance in deciles, for which 34-43 points are available; and academic achievements, which are worth up to seven points.
All applicants to the foundation programme (including the military and the academic foundation programme) are required to take the SJT on one of two national dates under invigilated conditions at their medical school—or at an arranged UK venue for those applying through the Eligibility Office. The SJT paper comprises 70 items that must be answered in two hours and 20 minutes. There are two formats for the questions: rank five possible responses in order, and select the three most appropriate responses. The points awarded will be between 0 and 50, with the top scoring applicant receiving 50 and the bottom scoring candidate receiving 0. The SJT scores in the pilots were normally distributed, which, alongside the EPM, gave a spread of total points that allowed separation of applicants on merit.
How applicants can prepare
Early on, applicants should find out how the EPM deciles’ process works in their school. Applicants also need to have copies of verifiable evidence for any additional academic achievements.
The SJT is a test of aptitude and judgment, not clinical knowledge, so it is not possible to revise. There are clear overlaps between what will be tested in the SJT and the activities linked to the personal and professional development themes within MBBS and MBChB degrees, so students will benefit from engaging with these. Applicants should also gain experience of the daily work of a foundation doctor.
In addition, the SJT is an opportunity to demonstrate what skills have been acquired within and outside university—skills such as leadership, decision making, and working as part of a team. A key part of preparation will be familiarisation with the attributes in the national person specification and the guidance in Good Medical Practice.8
The Improving Selection to the Foundation Programme website (www.isfp.org.uk) has a wealth of documentation about the work to develop the SJT and EPM. In addition, there is a downloadable monograph explaining SJTs and a practice paper on the Foundation Programme Office’s website (www.foundationprogramme.nhs.uk). This practice test includes items linked to all the key attributes underpinning the SJT and provides answer keys and explanations.9 An applicant should do this practice test under timed conditions to get used to the SJT format and should pace their responses to complete the paper (box 2).
Box 2: Top tips for the SJT
Read the question carefully and in full before answering the item
Completely answer all items—remember that there is no negative marking
Use only the information provided in the item—do not make assumptions
Draw on your clinical experience and the teaching and learning about professionalism in Good Medical Practice8
Answer options are discrete and should not be thought of as chronological (that is, do this, then this)
Remember that you are not being asked to judge whether an option is right or wrong
Also remember that you are being asked what you “should” do, not what you “would” do
Keep an eye on the clock—do not spend too long on any item. You should aim to spend around two minutes on each item and to complete the paper
Double check that you have marked the answers correctly on the answer sheet
The SJT and EPM offer considerable improvements in the short and long term for applications to the foundation programme, including fairness, transparency, validity, reliability, more effective use of clinicians’ time, and longevity. The Foundation Programme Office is working closely with the Medical Schools Council to ensure that these benefits are delivered, and plans are in place for ongoing evaluation.
BMJ Careers Fair
Find out more about foundation programme selection at the BMJ Careers Fair on Friday 19 and Saturday 20 October at the Business Design Centre in Islington, London. Register online at careersfair.bmj.com.
Competing interests: None declared.