BMJ  2007;334:543-544 (17 March), doi:10.1136/bmj.39154.476956.BE (published 14 March 2007)

Editorials

Why the UK's Medical Training Application Service failed

No convincing validation of the new process was provided

Stung by near universal condemnation of its new process for short listing junior doctors for specialty training posts,¹ the UK government acted. On Tuesday 6 March it announced a review and by Friday 9 March it had accepted the review's first round of recommendations.

These recommendations were that the first round of interviews should continue as planned, but with a "strengthened" interview process. Applicants not short listed for interview can have their application form reviewed by a trained adviser, which might result in the offer of a first round interview. Applicants could now include CVs and portfolios to support their application.

The review also promised major changes to the second round, including changes to the application form and the scoring system. The revised approach will be fully tested and agreed with stakeholders before it is introduced.² The review expects to make its final report by the end of the month.

While the response has been commendably fast, it begs the question as to why if the flaws in the new proposals were so easily spotted—and rectified—they weren't noticed earlier. All parties to the hastily convened review—the government, the royal colleges, and the British Medical Association—were represented during the more leisurely deliberations over the original proposals.

Yet this is an episode with many more questions than answers, and deciding how to apportion blame will have to wait until the system is fixed. In the meantime, we are stuck in the middle of a process, with no final outcome to evaluate, awash with anecdote.

That some of the best and brightest of their generation of junior doctors had not been short listed for interview was cited as incontrovertible evidence that the new process needed fixing. Although numerous anecdotes support this, others support the opposite, with senior doctors believing that, "among their juniors, the best are those with most interviews and those with no offers are the least able."³

Undoubtedly, there were technical problems with processing so many applications, especially as a big bang computerised approach was preferred over a few closely observed pilots. Some people doing the short listing were insufficiently trained. Short listing timescales were absurdly short and coincided with half term holidays (just as the interviews will conflict with Easter holidays).

Short listers saw the responses to a single question, not in the context of the rest of the form. Blind to previous employment history, they could not act on the premise that the best predictor of future performance is past performance. Answers to a series of hypothetical questions about clinical practice were weighted more heavily than verifiable, relevant achievements (thereby reducing the incentives for future doctors to work towards them).

The most serious charge against the new system is that it apparently lacked any validation. Did the application form ask about the sorts of things we consider relevant? Did it cover all aspects we want to measure? Was the overall score related to other variables in the way we would expect? Was the assessment repeatable and sufficiently objective to give similar results for different observers?⁴

We don't know—but we needed to know before the old system was jettisoned. Unfortunately, the process was shrouded in the utmost secrecy—even the questions and the scoring system were kept under wraps for as long as possible. (One short lister filed a bogus application just to get a look at the questions.)

The review is changing all this. It has recommended that ratios of numbers of applicants to jobs should be made available by specialty, entry level, and geography. It might also recommend the publication of the numbers of interviews offered to each successful applicant. If the anecdotes are correct, and the "best" candidates are being offered four, then deaneries can better calculate how many interviews they will need to offer to fill all their available positions.

To know how to react to claims that there are 8000 more applicants than jobs we need to know whether there are more, fewer, or the same number of jobs as beforehand, and where the doctors come from (the United Kingdom, the European Economic Area, or elsewhere). Is the process being "swamped" by applications from non-UK doctors? This is the elephant in the room, which no one except the international medical graduates themselves seem ready to talk about.⁵ Will the final appointments be made totally on merit, they wonder, or will the interviewing panel feel compromised by the fact that UK taxpayers have contributed £250 000 (370 000; $480 000) towards training the not quite so good UK doctor in front of them?

It's early days, but it looks like the review is favouring a return to what existed before—presumably on the grounds that it was tried and tested. But the little testing that has been done suggested that the old ways had their own biases. Yet the recent past is already being constructed as a golden age, with everyone getting the job in the specialty they wanted, in the region they wanted, and with spouses being welcomed into jobs in the same deanery—which is of course nonsense.

Coincidentally, this is the week when the Match (the United States' annual scheme for matching medical graduates to residency programmes) releases its results, something that has been happening without much rancour since 1952. Closer to home, UK general practitioners have devised machine readable tests (sat by all applicants on the same day) as the gateway to selection interviews.

For the time being, UK junior hospital doctors and those who administer the process for their selection into training positions may feel too traumatised to look forward to any radically new proposals. But they might look sideways at how other countries (and other doctors within their own country) manage to fit applicants to positions. They should avoid looking back.

BMJ, London WC1H 9JR

---

Competing interests: My wife, who is an obstetrician and gynaecologist, has short listed and will be interviewing ST1 candidates for the London Deanery.

Provenance and peer review: Commissioned; not externally peer reviewed.

References

1. Coombes R. How specialist training reform sparked crisis of confidence. BMJ 2007:334:508-9.
2. Department of Health. Review of the Medical Training Applications Service selection process—government response to concerns. Press release 10 March 2007. www.gnn.gov.uk/environment/fullDetail.asp?ReleaseID=270216&NewsAreaID=2&NavigatedFromDepartment=False
3. Eccles SJA. In defence of MMC and MTAS. Rapid response to Coombes R. How specialist training reform sparked crisis of confidence. BMJ 2007;334:508-9. www.bmj.com/cgi/eletters/334/7592/508#162034[Free Full Text]
4. Bland JM, Altman DG. Validating scales and indexes. BMJ 2002:324:606-7.
5. Rapid responses to: Coombes R. How specialist training reform sparked crisis of confidence. BMJ 2007;334:508-9. www.bmj.com/cgi/eletters/334/7592/508#162034[Free Full Text]

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Related Articles

Modernising Medical Careers laid bare

Tony Delamothe
BMJ 2007 335: 733-734. [Extract] [Full Text] [PDF]

Selection for specialist training: what can we learn from other countries?

Tony Jefferis
BMJ 2007 334: 1302-1304. [Extract] [Full Text] [PDF]

Real impact

Trish Groves
BMJ 2007 334: 0. [Extract] [Full Text]

How specialist training reform sparked crisis of confidence

Rebecca Coombes
BMJ 2007 334: 508-509. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

• Fuller, G. (2007). Revolutions in neurological training. PN 7: 356-359 [Full text]  
• Delamothe, T. (2007). Modernising Medical Careers laid bare. BMJ 335: 733-734 [Full text]  
• Jefferis, T. (2007). Selection for specialist training: what can we learn from other countries?. BMJ 334: 1302-1304 [Full text]  

Rapid Responses:

Read all Rapid Responses

MMC: Words speak louder than Actions

David Owen, et al.
bmj.com, 17 Mar 2007 [Full text]

Any ethics (or evidence) applied to MTAS?

Jane L Gibbins
bmj.com, 17 Mar 2007 [Full text]

Learning from Primary Care

Anne Holmes
bmj.com, 17 Mar 2007 [Full text]

The Caldwell Solution to MTAS

Gordon Caldwell
bmj.com, 18 Mar 2007 [Full text]

Pilot project would have been more appropriate

Mahesh C Thagadur
bmj.com, 19 Mar 2007 [Full text]

MTAS is not the first

Alison Giles
bmj.com, 19 Mar 2007 [Full text]

Trust Me I'm a Doctor ? - MTAS - The Problem and a Solution

Martyn W Neil
bmj.com, 19 Mar 2007 [Full text]

Facing our shame and finding our voices again.

Eileen McGinley
bmj.com, 20 Mar 2007 [Full text]

A countdown to confrontation

Ebaa SS Alozairi
bmj.com, 20 Mar 2007 [Full text]

I was on the RemedyUK March

Annie Y Lau
bmj.com, 21 Mar 2007 [Full text]

Many people missing the point.

Katrina S Davis
bmj.com, 21 Mar 2007 [Full text]

A bleak future for Postgraduate Training

Satyajit Nag
bmj.com, 21 Mar 2007 [Full text]

raging against MTAS

clarissa d fabre
bmj.com, 23 Mar 2007 [Full text]

An Orwellian Mess

Swethajit Biswas
bmj.com, 23 Mar 2007 [Full text]

Thank you for your support, but will the government deliver?

Abhijeeth R Shetty
bmj.com, 23 Mar 2007 [Full text]

MTAS. Focus the aetiology not the symptoms

Anthony E Young
bmj.com, 29 Mar 2007 [Full text]

Who is responsible for MTAS?

Chris M Laing
bmj.com, 31 Mar 2007 [Full text]

Critical week for MMC/MTAS: Individual Consultants and Staff-Councils will decide the fate of MTAS

Morris J Brown, et al.
bmj.com, 2 Apr 2007 [Full text]

Ways of improving

Jason Y K Chan
bmj.com, 3 Apr 2007 [Full text]

Re: Critical week for MMC/MTAS: Individual Consultants and Staff-Councils will decide the fate of MTAS

Gregory W Taylor
bmj.com, 3 Apr 2007 [Full text]