Rapid Responses to:
|
|
Rapid Responses: Rapid Responses published:
-
![[Read Rapid Response]](/icons/misc/sectionDOWN.gif)
Maintenance of Professional Standards
- Kieran A Fernando (24 September 2007)
-
Selection methodology; fact, fiction and the future
- Fiona Patterson (4 October 2007)
-
More light less heat
- Frank R Smith (11 October 2007)
-
Re: MTAS or a tale of evidence heedless medicine
- Alison S Carr (11 October 2007)
-
Re: Selection methodology; fact, fiction and the future
- Matthew J Daniels (11 October 2007)
-
Re: Selection methodology; fact, fiction and the future
- Alison L Gill (11 October 2007)
-
Evidence heedless medicine
- John Sanderson (11 October 2007)
-
No defence
- ben dean (11 October 2007)
-
Unsatisfactory response by Ms Patterson
- Peter von Kaehne (12 October 2007)
-
Selection methodology: more fiction than fact, and a worrying future
- A Thomson (12 October 2007)
-
Selection methodology: more fiction than fact, and a worrying future
- Peter Szatmary (19 October 2007)
-
MTAS slides
- Chris M Laing (19 October 2007)
|
|
|||
|
Kieran A Fernando, ST2 Core Medical Training West Midlands B15 2TH
Send response to journal:
|
I’m in complete agreement with Parashkev Nachev’s views (BMJ 2007;335:615) on the medical training application system (MTAS) and concerns surrounding the maintenance of professional standards in specialist medicine. I was fortunate enough to have obtained a training post within the MTAS/Modernising Medical Careers(MMC) debacle, at specialty training (ST)2 level in Core Medical Training. My issue of concern is just exactly what does ‘core medical’ training comprise of? So far, in my ST2 post, my typical daily routine consists of writing in patient notes, completing and delivering numerous radiology request forms, phlebotomy, and carry out practical procedures – usually the most invasive of which is intravenous cannulation. In addition, the whole of my ST2 year does not, at any point contain acute medical on-calls! This is appalling. Surely the ‘core medical training’ skills we are required to obtain are mainly seen on such on- calls – treating heart attacks, heart failure, exacerbations of chronic lung disease, pneumonias, strokes etc., etc. I feel very fortunate to have completed a traditional 2½ year medical SHO rotation prior to my current post, and feel competent in managing the vast majority of acute medical presentations. This rotation provided me with more than the ‘core medical’ skills. I am concerned with regards to my junior medical colleagues, who have completed their foundation years 1 and 2 and now entering ST schemes. I fear that they will not have sufficient exposure to acute medicine and will not become confident in decision-making and managing medical emergencies. I agree with BMJ Editor Fiona Godlee’s views (Editor’s Choice BMJ 22 Sep 2007) that we are increasingly likely to see the words ‘await senior review’ in the clerking notes of junior doctors. Furthermore I fear that if changes are not made to by MTAS/MMC, there will be increasing concern regarding the maintenance of professional standards in specialist medicine. Kieran Fernando MRCP Competing interests: None declared |
|||
|
|
|||
|
Fiona Patterson, Professor of Organisational Psychology and Partner Work Psychology Partnership City University, London. EC1V 0HB
Send response to journal:
|
This response has been amended on legal advice.
Although there is over a century’s literature on selection methodologies, rarely has any process provoked such fierce debate as MTAS. The problems surrounding selection methodology are multi- faceted and multi-dimensional, well beyond disregarding CVs, relying on ‘white space’ questions and poor IT delivery. For the distress caused, I share the anger of the profession. The inquiry led by Sir John Tooke details the facts about our role in the process (http://www.mmcinquiry.org.uk). Here, I highlight critical issues to encourage close scrutiny of facts surrounding principles, methodology and context underlying MTAS, so there is learning for the future. Clearly, in the past, selection practices in medicine have been effective. Although few would deny there is scope for improvement, the CV and interview process has generally worked well. So why change? MTAS was devised alongside the MMC programme introduced by the Department of Health, where the fundamental principles underpinning gateways to progression were changed overnight. Consequently this changed the rules governing selection. Traditionally, medicine has relied on robust CV indicators of attainment such as work experience and College exams for selection decisions. The MMC pathway relied on the belief that doctors could be selected to specialities without any prior experience in that specialty. The selection methodology in MTAS was designed for ST1, not for thousands of doctors already working in specialties. Under MMC principles and in compliance with PMETB (http://www.pmetb.org.uk/index.php?id=456), we were advised that work experience and exams could not be scored, or used to rank applicants. The introduction of run-through ST1 posts was new. I have worked on selection methodology in medicine for over 12 years. This work informed selection centre development and the introduction of new shortlisting tests in General Practice (GP) 1-3. In 2007, all deaneries worked together through a GP national office, where thousands of doctors are successfully appointed using this process 4. Since 2002, in partnership with doctors we developed selection methodology for many secondary care specialties 5-7 and for graduate entry into medical school 8. Although commonality exists across all specialities and levels, selection criteria for each are distinct, with evidence supporting different priorities between specialties. Having completed this work, in 2004, I was invited to meet the MMC team to advise on selection methodology into specialty training. I recommended developing a national test for shortlisting (supported by early evidence from GP) and validated selection centres with full College involvement and large-scale consultation. Following this meeting, I received no further correspondence from the MMC team and no pilots were put in place. In May 2006, we won an open competition tender organised by the Department of Health. Our work included advising on selection methodology for Foundation programmes and the GP selection process. For specialty selection, the scope of work states; “The number of applicants expected to apply for entry into Specialty Training is approximately 6,000 and that applications will be via a single electronic national portal entry system (separate project) the working assumption for the closing date will be 5th January 2007.” At the outset we were asked to advise on selection methodology for ST1. We were not asked to deliver selection methodology for doctors in ‘transition’ via ST2, ST3, ST4 and FTSTAs, nor academic posts. We believed these arrangements would be delivered via local processes. The rules and parameters governing MTAS were defined by MMC, based on PMETB principles and via the COPMeD steering group, represented all stakeholders. Given the time scale (less than 16 weeks) there was no option but to use materials from existing application forms used (over several years) for entry into specialist training. By contrast, in collaboration with the GP national office, my team designed the shortlisting test with GPs, which has shown to work well. For the future, the GP model has been identified as best practice. However, this model cannot be transferred into all specialties. Medicine is a broad discipline and secondary care is significantly different requiring bespoke selection methodologies. There is added complexity due to different selection ratios for both specialties and locations. ‘One size’ cannot fit all. Some believe selection practices in other professions can be readily transferred but medicine, in the UK, is truly unique. For those deciding policy, few understand what a clinician does on a daily basis. A significant challenge is to translate the needs of the profession to policy makers. I have advised on selection methodology in all sectors. In the past 12 years I have learned more from collaborating with the medical profession than from any other. Unlike selection approaches used by some organisations, I applaud the focus on psychometric scrutiny, the need for validatory evidence and the demand to treat human beings with respect and dignity in the process. The fact is, MTAS was not designed by psychologists. Without a full understanding of the issues, we cannot hope to navigate the future, which looks yet more challenging. References 1. Patterson F, Ferguson E, Lane PW, Farrell K, Martlew J, Wells AA. Competency model for general practice: implications for selection, training and development. Br J Gen Pract 2000;50:188-93. 2. Patterson, F., Lane, P., Ferguson, E. & Norfolk, T. A competency based selection system for GP trainees. BMJ 2001 323, 2. 3. Patterson F, Ferguson E, Norfolk T, Lane P. A new selection system to recruit general practice registrars: Preliminary findings from a validation study. BMJ 2005;330:711-4. 4. Plint, S., Gregory, S., Evans, G. (2007). Recruitment to GP specialty training 2007. BMJ Career Focus 335: gp73-gp75 5. Randall R, Davies H, Patterson F, Farrell K. Selecting doctors for postgraduate training in paediatrics using a competency based assessment centre. Arch Dis Child 2006; 91:444-8. 6. Randall R, Stewart P, Farrell K, Patterson F. Using an assessment centre to select doctors for postgraduate training in obstetrics and gynaecology. The Ostetrician and Gynaecologist 2006;8:257-62. 7. Rowley D, Patterson F. The right choice: A pilot selection centre to improve selection of future surgeons. Surgeons News, October 2007. 8. Kidd J, Fuller J, Patterson F, Carter Y. Selection Centres: Initial description of a collaborative pilot project. Proceedings for the Association for Medical Education in Europe (AMEE) Conference, September 2006, Genoa Italy. Competing interests: I am a Partner in the Work Psychology Partnership who were awarded a contract by the Department of Health to advise in selection methodology in June 2006. We were investigated for competing interests for earlier publications but no accusation was upheld, including that from the BMJ. |
|||
|
|
|||
|
Frank R Smith, Primary Care Taskforce Lead South Central SHA Highcroft Winchester SO22 5DH
Send response to journal:
|
The BMJ of the 22nd September has a number of pieces on medical training.
The Editor has 'pondered the BMJ's coverage' but she should re-assess her sanctioning of Nachev's personal view as completely counter-productive to the debate, despite its tabloid appeal in playing to the masses of (deservedly)unhappy junior doctors.
Selection science is not an oxymoron, and whilst longitudinal studies still need to be done, there is evidence building of the utility of different selection methods compared to the traditional CV and interview.
The Tooke analysis of the events of 2007 is likely to identify some key learning points. The BMJ should aim for more light but not heat in this debate.
Competing interests: Have worked on developing selection for GP with Professor Patterson |
|||
|
|
|||
|
Alison S Carr, Deputy Postgraduate Dean, NHS Education South West (Peninsula Institute) Plymouth PL6 8DH
Send response to journal:
|
Dear Editor I would like to correct some of the details that have been cited by Dr Nachev on the development of resources for recruitment and selection into specialty training in 2007. The author suggests that the criteria and procedure for selection in MTAS were principally designed by a handful of organisational Psychologists from Work Psychology Partnership and that the selection methods developed have never been used to select specialist trainees. Neither of these suggestions are correct. Professor Fiona Patterson and her colleagues from Work Psychology Partnership have worked alongside the medical profession for over 12 years in helping develop recruitment and selection methodology for recruiting specialist trainees such as General Practitioners, Obstetricians, Paediatricians, and Surgeons. This team specialise in recruitment and selection methodology and have applied their knowledge base to medicine in liaison with specialists from the medical specialties. Work Psychology Partnership have worked with GPs for over ten years in developing the recruitment and selection processes used successfully for recruitment into general practice training. In addition, for several years they have worked developing and evaluating recruitment and selection pilots into surgery with the Royal College of Surgeons. In fact almost all of the research published on recruitment and selection into medical training has been published with Professor Fiona Patterson as one of the authors. In this article, Dr Nachev remarks that every slide of the material prepared for the Department of Health he had seen was emblazoned with Work Psychology Partnership logo (www.mmc360.com/documents/ recruitment_to_specialist_training.pdf). In fact most of these slides were designed by myself in my role as Honorary Associate Dean for the National Recruitment and Selection Project 2007. In this role, I was one of three doctors who accompanied the methodology team on Deanery roadshows around the United Kingdom providing information for Deanery staff and trainers on the processes of recruitment and selection into specialty training proposed for 2007. In addition, it must be stressed that the medical input to the recruitment and selection process was provided by doctors and that processes used were introduced with consultation of the Royal Colleges, Work Psychology Partnership acting as Consultants in recruitment and selection methodology. The criteria for recruitment and selection into Specialty Training, from which the methodology was developed, were those as laid down by the PMETB (http://www.pmetb.org.uk/index.php?id=456). Competing interests: None declared |
|||
|
|
|||
|
Matthew J Daniels, FTSTA ST2 CMT Addenbrookes Hospital, CB2 0QQ
Send response to journal:
|
In Professor Patterson statement is clear that her expectations for the process and the reality of its implementation were quite at odds. Why then has it taken until October 2007 for these concerns to be voiced? I recall one of the white box questions in the probity section - "Give a specific example of a time when you became aware that a clinical mistake had been made, either by you or someone else. How did you deal with this situation and how did your actions contribute to the outcome?" As the Fidelio group have already reminded us; "All that is necessary for the triumph of evil is that good men do nothing."1. 1 Brown M et al The Lancet 2007; 369:967-968 Competing interests: Dr in training disillusioned by the whole process |
|||
|
|
|||
|
Alison L Gill, ST2 Medicine Harrogate District HospitalHG2
Send response to journal:
|
Ms Patterson's points would be far more credible were it not for the fact that seven of her eight references were self-citations! She accepts that pre-existing "selection practices in medicine have been effective", and that few "understand what a clinician does on a daily basis". What qualifications then does she have to make such recommendations and changes to medical selection and training? She reports having "learned more from collaborating with the medical profession than from any other" - so why during this shambolic recruitment system, did she not think it necessary to consult with exactly those people that were to be affected by the changes? Competing interests: MTAS applicant, four interviews, offered two posts, only to find months down the line that I should have been offered all four but due to a "system error" I was listed as having accepted an offer before I was even made aware of it! |
|||
|
|
|||
|
John Sanderson, Professor of Clinical Cardiology B16 8AH
Send response to journal:
|
Belatedly I have just read Dr Nachev's brilliant analysis of the failure of MTAS. The lack of any experimental evidence is typical of most social or health economics policy. The contrast between the introduction of a new medicine and a new administrative 'therapy' or reorganization could not be greater. It appears that major NHS changes are introduced based on no-more than anecdotal evidence which would not be tolerated in the realm of medical therapeutics. Why are not the same standards applied and a proper controlled trial done of some of the proposed changes? It is not too difficult to envisage two differing policies being tested in two health care regions and the results tested after 5 years like in a clinical trial. The RCT has been one of the greatest steps in medicine and as we all know the results of a large clinical trial are often the exact opposite of the expected, obvious or 'logical' conclusion. Massive social and administrative changes are often introduced on a whim and a feeling that it must be right. The same mistakes are about to be made by Darzi and his collegues in the government with respect to general practice and polyclinics. No doubt millions of pounds more will be wasted on major stuctural changes with zero evidence of any actual benefit. Competing interests: None declared |
|||
|
|
|||
|
ben dean, sho oxford
Send response to journal:
|
Fiona Patterson is clearly attempting to distance herself from the complete and utter failure of MTAS 2007 with her above response that lists many references that are used as so called 'evidence'. The process that she had a large part in creating and forcing upon us was certainly not suitable for the selection of any trainees; irrespective of whether they were Foundation trainees, ST1s, ST9s or monkeys. I have not managed to find a single Foundation trainee who thinks that their selection process over recent years has been anything other than a load of politically correct hogwash. The MTAS process and the use of white space questions were proven beyond any doubt to be rubbish of the highest order, and certainly not fit for use in any selection process. If Prof Patterson wants evidence, then I think the year 2007 provides an overwhelming quantity of evidence that should should force her to go back to the proverbial drawing board before inflicting any more of this upon unsuspecting juniors. Undoubtedly Prof Patterson was only one of many of a dysfunctional heirarchy that was to blame, however she is a little naive if she thinks that she can talk her way out of any responsibility so easily. The future is indeed challenging and this is because so many people including Prof Patterson did so much to create a completely useless selection process last year. I suggest if this argument becomes about evidence, then the events of 2007 can provide more concrete evidence than any number of psychoeducationalist trials. Competing interests: None declared |
|||
|
|
|||
|
Peter von Kaehne, General Practitioner Scotland
Send response to journal:
|
Although there is over a century’s literature on selection methodologies, [...] This could suggest that there are heaps and heaps of data available on the subject, that Ms Patterson's role was only one of sifting the abundant and overwhelming evidence and that the medical profession was grossly negligent in its previous blatant disregard of good science on the subject. And yet, all but one quote in Ms Patterson's response appear to be self references. What does this tell us? While I am in no position to judge even remotely how close and how responsible Ms Patterson's outfit was, the response by Ms Patterson leaves me in little doubt that she was too close and had too much responsibility. So, please leave us alone with your suggestion that you share the anger of the profession. Chances are, that you are a perfectly valid target for the anger of the profession. Your letter at least does not reassure me on that account. Competing interests: None declared |
|||
|
|
|||
|
A Thomson, Doctor London
Send response to journal:
|
Although Prof Patterson claims that "there is over a century’s literature on selection methodologies," it is disappointing that 7 out of 8 of her references are self-citations. Prof Patterson fails to address the question "why change?" other than stating that the reasons for change rested on the potentially flawed belief that doctors should be forced to choose a speciality with no prior relevant experience. She neither questions nor attempts to justify the validity of such a significant assumption - how can she then justify any process which is based on it? Her account of the development process, if true, is very worrying - although she attempts to distance herself from the disaster, she does admit her involvement, with the admission that selection forms for entry above ST1 were hastily cobbled together "from existing application forms", that there is no evidence of their validity outside of GP selection. Prof Patterson was aware of the flaws, and could have used her authority as an expert in selection methodology to halt this sorry process which wasted so much time. She could have objected, refused to allow her work to be implemented in this way and firmly recommended continuing with current selection procedures. Why did she do none of these things? Why did she press ahead with an enormous human experiment for which no ethical approval had been sought or granted, and to which the subjects had not consented. Her reply addresses none of these questions, and I find it rather sinister that she is already looking to the future without attempting to learn from the terrible mistakes in her recent past. Competing interests: Dr Thomson was one of the thousands of UK doctors who were sacked and made to reapply for their own jobs using flawed selection methods which Prof Patterson and the Work Psychology Partnership helped to develop. |
|||
|
|
|||
|
Peter Szatmary, SHO orthopaedics Hamilton, NZ
Send response to journal:
|
Medical education in general and selection into medical specialties in particular are a fairly recent field of interest and Prof Patterson is indeed the expert in the field. The reason that she self references extensively is most likely a reflection of the fact that she has indeed published a large proportion of the available work. However, that in itself together with several of her own admissions - ie. that most of her work relates to GP selection, that none of it had been piloted and that she was forced to use material designed for GP selection for other specialties while accepting that may not be a valid approach - means that she should have intervened to stop the process going ahead as it did. Speaking for surgery in particular, I am aware of at least two groups (University of Toronto and Imperial College London) who have attempted to devise tools that might predict future success in surgery, but have so far only managed to show that surgical skill can be taught to any trainee and that practice increases proficiency. Whilst I can see that the idea of matching what are essentially personality tests (MTAS white space questions,NB) to future careers is scientifically intruiging, one has to remember that there is no prospective evidence to suggest that someone who scores 3 instead of 2 on the 'team-leader' question, or 3 instead of 4 on the 'coolness under pressure' question is in any way more or less suited to any medical or surgical specialty: especially when poor preparation means that inter-rater variability is likely to be high. Competing interests: UK trained surgical SHO working in NZ post MTAS |
|||
|
|
|||
|
Chris M Laing, SpR nephrology and ICM St Thomas's Hospital London
Send response to journal:
|
I'd like to congratulate Dr Nachev on his piece. I would like to contribute 2 points. (1) It may be necessary to perform aptitude/potential testing (such as that proposed in performance assessments centres) in the private sector. This is because the candidate may be entering a new line of work and, due to obvious sensitivities, it is often impossible to approach the candidates previous employer for an evaluation of their abilities. Neither is the case for junior doctors. They have been "in medicine" since the age of 18 and have undergone continuous assessment both at medical school and as junior doctors in prior posts. Why can't these assessments be more usefully graded (as in USMLE and board exams in the US) to save this nonsensical duplication of effort. Secondly, unlike in the private sector, the NHS is a monopoly employer of junior doctors. Surely it is entirely wasteful that it cannot obtain detailed references from prior colleagues who are trusted enough to work in the same organisation? This would be unlikely to happen if a candidate was planning to move departments in a commercial organisation. (2) Dr Nachev provided the following link to a series of MTAS slides used for briefing selectors www.mmc360.comdocumentsrecruitment_to_specialist_training.pdf These mind-boggling and almost hallucinogenic slides are one of the most badly designed sets I have ever seen. Perhaps I could refer the authors to the work of Edward Tufte (www.edwardtufte.com). In spite of his recognition as an international authority in visual communication he offers very useful advice on the educational limitations of powerpoint on his website for absolutely no charge. Tufte deplors the use of "chart junk" and branding logos on such slides. Alison Carr, as an educationalist, should be aware that such logos are distracting. Might I also ask why, if this is her presentation in her role as a postgraduate dean, her slides are branded with the logo of a private consultancy company? Competing interests: None declared |
|||