MTAS or a tale of evidence heedless medicineBMJ 2007; 335 doi: https://doi.org/10.1136/bmj.39342.515961.59 (Published 20 September 2007) Cite this as: BMJ 2007;335:615
- Parashkev Nachev, clinical research fellow, department of clinical neuroscience, Imperial College London
The greatest achievement of modern medicine is arguably not any single therapeutic advance, but the development of a highly sophisticated framework for distinguishing a true advance from what merely looks like one. Evidence based medicine has completely transformed our profession to the extent to which no doctor—not even the most cavalier one—would countenance a change to current practice that has not been justified by a rigorous comparison between the old and the new. What constitutes a rigorous comparison is well established; indeed, agreement on the principles of designing and reporting therapeutic trials is so widespread that all good journals refuse to publish any study that does not fit the standard template. These principles are simple and incontestable: all comparisons between treatments should be fairly made, with meticulous attention to potential sources of bias, with clear outcome measures, and with inferences limited to only what is statistically justified by the data.
One might therefore have thought that the response of any competent physician to what I am about to describe would be predictably derisive. Imagine that the government proposed a radically new treatment (let's call it Effupin) for a complex and important condition that has hitherto been treated in an imperfect but largely satisfactory way. Effupin's mode of action is unknown: its use is motivated by anecdotal reports from veterinary practice. It has never been tested with regard to any accepted primary outcome measures, and such evidence as exists in its favour comes from barely a handful of small open-label studies that look at indicators only speculatively related to outcomes. Indeed, its inherent plausibility and their own authority in the field are its proponents' main arguments for it, although they have never seen or treated the condition in question.
Imagine further that Effupin has been designed by a company that stands to benefit directly from its widespread adoption. The government has appointed the company on the basis of a process the details of which it refuses to make public. Finally, the government insists that Effupin is compulsory and that no clinician is individually allowed to use any alternative.
Now one would not have to be an expert in evidence based medicine to recognise the fatal flaws in such a proposal. Indeed, if a first year medical student were to find it acceptable he or she could justifiably be made to retake his exams. And yet this is precisely the kind of error that the leaders of our profession have committed. The error, in fact, is infinitely more important than the treatment of any one condition because it affects our ability to treat patients at all.
I am speaking, of course, of the United Kingdom's new Medical Training and Applications Service (MTAS) for the selection of specialist trainees. Not the website, or the technical glitches that have occupied such a disproportionate amount of print in the lay press, but the fundamental principles of selection on which it is based.
The criteria and procedure for selection in MTAS were principally designed by a handful of organisational psychologists engaged through their consulting firm, Work Psychology Partnership, for a fee of £92 950 (€134 000; $186 000) excluding value added tax (www.publications.parliament.uk/pa/cm200607/cmhansrd/cm070423/text/70423w0020.htm). The Department of Health has refused to reveal how they came to be appointed (a copy of the DoH's refusal to release this information under the Freedom of Information Act is available from the author). None of the known members of Work Psychology Partnership have any medical qualifications (www.workpsychologypartnership.com). The selection methods they have developed have never been used to select specialist trainees. The superiority of their methods is arbitrarily assumed—indeed their promotional literature suggests that the only substantive conceptual reason why doctors may object to them is a dispositional “resistance to change” (www.mmc360.com/documents/recruitment_to_specialist_training.pdf). Unsurprisingly, their claims are not supported by any scientific studies that examine the critical outcome measures—it could hardly be otherwise given that no such study can be carried out in less than the time it takes to train a specialist. Instead, we have a series of essentially anecdotal reports, citing favourable feedback from key “stakeholders.” That the authors do not discriminate between anecdote and evidence is obvious from the proposed selection process itself, in which the greatest weight is given not to demonstrable achievements, but to apocryphal tales from the applicant's clinical career.
If the evidence falls disastrously short of the standards to which we are accustomed, the ethics of its publication are in my view arguably kindred. Despite the obvious potential conflict of interest, Professor Fiona Patterson, apparently the principal agent of Work Psychology Partnership in this project, does not mention her consulting firm anywhere in the published reports, or on her academic website. By contrast, every slide of the material prepared for the Department of Health I have seen is emblazoned with the Work Psychology Partnership logo (www.mmc360.com/documents/recruitment_to_specialist_training.pdf).
In the hundreds of column inches that have been written about this disaster, little has been said about the role of the royal colleges in accepting the new scheme as a validly justified intervention. As I have demonstrated, the failure was not so much foreseeable as glaring—from the outset. And yet, the leaders of our profession failed to take action when there was still time to act. The maintenance of professional standards in specialist medicine is the responsibility of the royal colleges: what is their 500 year history otherwise for? What else do they exist for?
And let us be clear that the principal issue here is professional standards, not the welfare of junior doctors, as the BMA tends to present it. Monstrous though their loss is, the hundreds of excellent doctors unfairly denied a career in British medicine will find success abroad or in some alternative walk of life.
It is hard to comprehend how the royal colleges could have allowed this system to be implemented without any apparent resistance. Either they were coerced into it, or they behaved in a grossly incompetent manner by not intervening. If it was the former, then the colleges owe it to the past and future of medicine in this country to declare that they were coerced by the government, whatever the consequences might be. If it was the latter, then in my view they are clearly unfit to represent our profession.
The author is funded by the Wellcome Trust.