Why the UK's Medical Training Application Service failed

Stung by near universal condemnation of its new process for short listing junior doctors for specialty training posts,1 the UK government acted. On Tuesday 6 March it announced a review and by Friday 9 March it had accepted the review's first round of recommendations.

These recommendations were that the first round of interviews should continue as planned, but with a “strengthened” interview process. Applicants not short listed for interview can have their application form reviewed by a trained adviser, which might result in the offer of a first round interview. Applicants could now include CVs and portfolios to support their application.

The review also promised major changes to the second round, including changes to the application form and the scoring system. The revised approach will be fully tested and agreed with stakeholders before it is introduced.2 The review expects to make its final report by the end of the month.

While the response has been commendably fast, it begs the question as to why if the flaws in the new proposals were so easily spotted—and rectified—they weren't noticed earlier. All parties to the hastily convened review—the government, the royal colleges, and the British Medical Association—were represented during the more leisurely deliberations over the original proposals.

Yet this is an episode with many more questions than answers, and deciding how to apportion blame will have to wait until the system is fixed. In the meantime, we are stuck in the middle of a process, with no final outcome to evaluate, awash with anecdote.

That some of the best and brightest of their generation of junior doctors had not been short listed for interview was cited as incontrovertible evidence that the new process needed fixing. Although numerous anecdotes support this, others support the opposite, with senior doctors believing that, “among their juniors, the best are those with most interviews and those with no offers are the least able.”3

Undoubtedly, there were technical problems with processing so many applications, especially as a big bang computerised approach was preferred over a few closely observed pilots. Some people doing the short listing were insufficiently trained. Short listing timescales were absurdly short and coincided with half term holidays (just as the interviews will conflict with Easter holidays).

Short listers saw the responses to a single question, not in the context of the rest of the form. Blind to previous employment history, they could not act on the premise that the best predictor of future performance is past performance. Answers to a series of hypothetical questions about clinical practice were weighted more heavily than verifiable, relevant achievements (thereby reducing the incentives for future doctors to work towards them).

The most serious charge against the new system is that it apparently lacked any validation. Did the application form ask about the sorts of things we consider relevant? Did it cover all aspects we want to measure? Was the overall score related to other variables in the way we would expect? Was the assessment repeatable and sufficiently objective to give similar results for different observers?4

We don't know—but we needed to know before the old system was jettisoned. Unfortunately, the process was shrouded in the utmost secrecy—even the questions and the scoring system were kept under wraps for as long as possible. (One short lister filed a bogus application just to get a look at the questions.)

The review is changing all this. It has recommended that ratios of numbers of applicants to jobs should be made available by specialty, entry level, and geography. It might also recommend the publication of the numbers of interviews offered to each successful applicant. If the anecdotes are correct, and the “best” candidates are being offered four, then deaneries can better calculate how many interviews they will need to offer to fill all their available positions.

To know how to react to claims that there are 8000 more applicants than jobs we need to know whether there are more, fewer, or the same number of jobs as beforehand, and where the doctors come from (the United Kingdom, the European Economic Area, or elsewhere). Is the process being “swamped” by applications from non-UK doctors? This is the elephant in the room, which no one except the international medical graduates themselves seem ready to talk about.5 Will the final appointments be made totally on merit, they wonder, or will the interviewing panel feel compromised by the fact that UK taxpayers have contributed £250 000 (€370 000; $480 000) towards training the not quite so good UK doctor in front of them?

It's early days, but it looks like the review is favouring a return to what existed before—presumably on the grounds that it was tried and tested. But the little testing that has been done suggested that the old ways had their own biases. Yet the recent past is already being constructed as a golden age, with everyone getting the job in the specialty they wanted, in the region they wanted, and with spouses being welcomed into jobs in the same deanery—which is of course nonsense.

Coincidentally, this is the week when the Match (the United States' annual scheme for matching medical graduates to residency programmes) releases its results, something that has been happening without much rancour since 1952. Closer to home, UK general practitioners have devised machine readable tests (sat by all applicants on the same day) as the gateway to selection interviews.

For the time being, UK junior hospital doctors and those who administer the process for their selection into training positions may feel too traumatised to look forward to any radically new proposals. But they might look sideways at how other countries (and other doctors within their own country) manage to fit applicants to positions. They should avoid looking back.