Rapid Responses to:

EDITORIALS: Celia A Brown and Richard J Lilford Selecting medical students BMJ 2008; 336: 786 [Full text]

Rapid Responses: Submit a response to this article

Rapid Responses published:

Tests of cognitive ability: Transparency and validity

George A Khoury   (12 April 2008)

Tests for what purpose?

Umar A Ahmad   (12 April 2008)

Cognitive tests are only one possible criterion

Jean McKendree   (16 April 2008)

Medical Student Selection at Entry Level

Urban J.A D'Souza   (17 April 2008)

Self-fulfilling Prophecy

Peter D Singleton   (18 April 2008)

Selecting Medical Students

Barry N J Walters   (1 May 2008)

Re: Self-fulfilling Prophecy

Christopher M Rayner   (14 May 2008)

Tests of cognitive ability: Transparency and validity 12 April 2008
George A Khoury, Consultant Surgeon Conquest Hospital TN37 7RD Send response to journal: Re: Tests of cognitive ability: Transparency and validity	Cognitive ability, humanity and diligence are identified as attributes of potential doctors but it is claimed that only cognitive ability can be assessed with reasonable accuracy. The Schwartz report (1) has previously recognised these challenges and recommended a ‘holistic assessment’ of all relevant factors, based on transparency and the use of assessment methods that are reliable and valid. The Supporting Professionalism in Admissions (SPA) report (2) on admission tests followed, with emphasis on rigorous validation. There are two distinct issues. The first is that of validity of admission tests, and the second is that of unfair non-transparent use of tests that are currently being tested. Aptitude tests such as BMAT and UKCAT based mainly on cognitive tests have not been rigorously validated. Evidence for BMAT is weak. The correlation between test and first year clinical examination performance was limited to section 2 and strong for veterinary and not medical students (3). Until the ‘tests have been tested’, a single section of an aptitude test disregarding a high overall score should not constitute a reason for rejection. Aptitude tests are being incorrectly applied as a short- listing tool in a non-transparent way independent of all other relevant criteria, against the code of practice of the Quality Assurance Agency for Higher Education and SPA recommendations. As to the validity of interview, the predictive accuracy for interview was low for academic ability alone in the reference cited by the editorial, but not low for clinical success. Interview remains important in the selection process to screen for non-academic core factors. Academic ability can be established from A level grades or marks attained in respective modules. The BMJ related articles lists for both this editorial and the previous feature on UKCAT (4) should ideally have included the recent BMJ Group newsletter about its acquisition of onexamination.com, or a declaration of competing interest. The website offers a UKCAT online revision course to medical school applicants at Ł50.00. This fee disproportionately exceeds that of other courses offered to qualified employed doctors, (FRCR Part 1, MRCP Psych 1 and 2, MRCGP and PLAB). Revision courses are already available on ‘how to do well at UKCAT’, increasing the divide between the affluent and less affluent (5). It appears that this is condoned by the BMJ Group, a subsidiary of the BMA, even though there is no validity to UKCAT (4). (1) Schwartz Report. Admissions to Higher Education: Fair Admissions to Higher Education Recommendations for Good Practice. September 2004. www.admissions-review.org.uk (2) Supporting Professionalism in Admissions. SPA Report on Admissions Tests used by Higher Education Institutions. June 2007 www.spa.ac.uk (3) Cambridge Assessment. A Report on the Predictive Validity of the BMAT (2005) for First Year Examination Performance on the Medicine and Veterinary Medicine Courses at the University of Cambridge. September 2007. Emery J L www.bmat.org.uk (4) UKCAT Among the Pigeons, Cassidy J. BMJ 2008 336: 691-692. (5) Undermining Access, Samuel DG. BMJ letters 2008 336:788 Competing interests: None declared
Tests for what purpose? 12 April 2008
Umar A Ahmad, 4th yr medical student University of Bristol, BS8 1TH Send response to journal: Re: Tests for what purpose?	After reading your article it got me thinking whether we actually need any new or any tests beyond the traditional interview to select students for medical school. If we are not satisfied with the current doctors we have as they are lacking some desirable trait or indeed possess some undesirable traits then this should prove suffecint for looking at test for entry into medical school. If however like me you are happy with the current calibre of the doctors then there seems to be little point in employing additional tests! Competing interests: None declared
Cognitive tests are only one possible criterion 16 April 2008
Jean McKendree, Senior Lecturer in Medical Education Hull York Medical School Send response to journal: Re: Cognitive tests are only one possible criterion	While it is true that academic measures, such as grades and A-levels, are the 'best' predictor of success in medical school, they are hardly great ones. In most studies, they can account for at most 25% of the variance in performance which leaves much unaccounted for. They are also better at predicting grades on written exams than they are clinical performance. In other words, ability to take tests (A-levels) predicts ability to take tests (medical school exams) rather than necessarily ability to be a good doctor. There are studies that suggest other measures that we might consider. For instance, several studies by Rest and colleagues have found that residents with high scores on measures of moral judgement had few low scores on clinical performance, while those with low scores on moral judgement were never among the highly rated clinical performers. The Rainbow Project at Tufts University under the direction of Prof Robert Sternberg uses measures of 'wisdom, intelligence and creativity, synthesised' (WICS) for admissions. They have found that academic performance increased, admissions of ethnic minorities rose by 30%, and they doubled the accuracy of their prediction of success compared to using the SAT alone (the SAT is much like the BMAT, a 'cognitive' measure). Sternberg has articles in Educational Leadership about the Tufts data and in the journal 'Intelligence' in 2006 about data from 15 colleges using a variation of the tests called the Sternberg Triarchic Abilities Test. The point, I think, is that any single measure of anything is rarely as good as a combination of relevant measures, especially when I would like to think that we want medical students who are strong in more areas than simply conventional academic ones. It is now a matter of working to find the best approach that isn't too costly in time and money. And what is needed for that is a decent level of funding to do the research and the courage to try something new. Competing interests: None declared
Medical Student Selection at Entry Level 17 April 2008
Urban J.A D'Souza, Associate Professor 88999, Kota Kinabalu, Sabah, Malaysia Send response to journal: Re: Medical Student Selection at Entry Level	Selection of students for medical graduate program may vary from county to country in the world. As mentioned in the editorial, ensuring equity, predicting human behavior, and defining characteristics of a ‘good doctor’ is a subject of constant interest. Producing a ‘holistic doctor’ with all the qualities of a normal human being is a major task of every university. As it is mentioned in the editorial cost of producing a doctor is about 200,000 pounds at the risk of greater wrong selection. Asian countries too spend a great amount of money in producing a doctor. Boom of private medical/university education is another lucrative factor for the non merit students to enter into the profession of doctors. Cognitive ability, humanity and diligence though the basic attributes a doctor requires, at the end of the training a majority of the product may miss the middle attribute which is an important factor while dealing with ailing patient. A patient cannot have a customer kind of approach by a doctor amidst his/her busy clinical practice, nor any patient can be dissected out wholly based on the ailment he suffer into different body parts. Holistic approach of medical practice by considering the human being as a whole is the basic requirement. In the general medical practice, most of the patients seek a doctor for empathy, that’s why many a time a ‘quack’ medical practitioner may be imparting a better impression/service even though his medical knowledge is minimal. Most of the modern qualified medical practioners may lack the art of medical practice but follow only the science of medicine. As rightly put by the father of modern medicine, Medicine is a science and art of healing. Psychological tests and communication skill tests at the entry level must be also a part of selection procedure at the entry level. Components of social sciences also may help in the curriculum of doctor of medicine program to get the deeper knowledge of society, socio-economic background factor while treating people of different background, ethnicities, and cultural differences. Age of entry level program is also another important criteria as many countires, student entering into the profession is between 17 to 18 years of age where, the maturity of a student may not be enough to select a medical profession progam where he has to hook the rest of his life. Atleast 24 to 25 years of age may be a right age to take a decision to enter into this great noble profession Competing interests: None declared
Self-fulfilling Prophecy 18 April 2008
Peter D Singleton, Principal Research Fellow University College London, N19 5LW Send response to journal: Re: Self-fulfilling Prophecy	Given the BMJ is primarily read by clinicians, we need to be aware of the risk of a self-selection here. Just as companies can develop an internal culture, which means that only similar personality types get recruited (archetypically McKinsey consultants), so the same happens with doctors. We select only those with high academic results, we train them in be problem-focused, and, not surprisingly, they will tend to select others with a similar profile. So there is a danger of complacency in Mr Ahmad’s approach in simply selecting what you already have as it may not be what you need. Historically, doctors needed good memories – they will probably continue to do so, but given that medical knowledge has easily out- stripped our ability to remember it all, we may do better with someone who is adept at using computer-based tools. We select people who relish solving problems and then train them in high-stress situations, so we select those that get a real buzz out of ‘winging it’ – but this may explain why medical practice is so slow to adopt care pathways and protocols as ‘it takes the fun out of medicine’. Belbin[1] showed that teams often fail because they miss the ‘completer/finisher’ person – we select clinicians to be people with the ‘shaper’ profile, who want to see their solution put into action (no wonder doctors only let patients talk for 17 seconds before interrupting them) – some of the most dysfunctional teams consist of a ‘shock of shapers’, all vying to get their idea through. Given we recognise that we need to move to a team-based approach to medicine and more flexible roles, then perhaps we really do need to review how we select people to become healthcare staff – and that may include whether they need to be ‘professionals’ and also whether they need to be managers rather than divas. I would admit to somewhat caricaturing clinicians here, but, having run focus groups across a range of stakeholder groups, clinicians are noticeably different in the way they approach matters. 1 R Meredith Belbin, Management Teams: Why They Succeed or Fail (Butterworth Heinemann, 2nd ed., 2004) Competing interests: None declared
Selecting Medical Students 1 May 2008
Barry N J Walters, Chair, Medicine Selection Committee, University of Notre Dame Australia, Fremantle Royal Perth Hospital, Perth, Western Australia 6847 Send response to journal: Re: Selecting Medical Students	Brown and Lilford's contribution, whilst thoughtful and insightful, includes some errors in fact, and omits some pragmatic concerns. The Graduate Australian Medical School Admission Test (GAMSAT) is not an aptitude test, but a rigorous and demanding knowledge based test in three parts - reasoning in humanities and social sciences, reasoning in biological and physical sciences, and written expression. Its intent is to ensure that applicants for medicine from a variety of tertiary backgrounds (all applicants must have a Bachelor's degree) demonstrate a high level of problem solving ability in other areas relevant to Medicine. For example, GAMSAT ensures that an applicant with a Law degree has achieved at least university entrance standard of competency in sciences, and that an applicant with a Physics degree displays a high level of ability in written expression and the humanities. By Brown and Lilford's criteria, GAMSAT is certainly a test of cognitive ability. The Australian Undergraduate Medicine and Health Sciences Admissions Test (UMAT), however, is indeed an "aptitude" test. It is "designed to assess general attributes and abilities considered important to the study and later practice of professions in the health sciences". It is not a knowledge based test and is enigmatic, idiosyncratic and byzantine in much of its detail. This test and not GAMSAT is worthy of Brown and Lilford's energetic criticism. The UMAT is a specious instrument, and it results in cruel disappointment for those applicants who are rejected. Their test result implies that they do not possess the "general attributes" necessary to study medicine. They cannot study to improve their result in this test. It assesses their innate attributes and finds them wanting. This is a harsh judgement on a young person leaving school, hoping for a career in a caring profession, and has led many, unfairly, to question their own worth. Brown and Lilford favour the use of school examination results alone for selection into medicine. This is indeed objective, but leads to selection of many candidates who do not complete the course. This high "attrition" rate was the reason most Australian medical schools abandoned academic scores as the only selection criterion. Further, using scores alone rejects students who score a fraction of a percent less than others, surely taking objectivity too far. Most graduate schools now use a multiphasic selection process, integrating previous university degree performance, GAMSAT score, and an interview, as surely we have a responsibility to see that a potential Doctor is at least able to converse freely and openly with those who might be patients sitting opposite their desk in the future. To be fair to each applicant and to the community, this is the only way to select medical students, despite its imperfections, and it is the pathway we have chosen. Ballots may have a role, but chance sometimes plays unexpected tricks on those who trust in her. Competing interests: None declared
Re: Self-fulfilling Prophecy 14 May 2008
Christopher M Rayner, GP Elstead GU8 6EG Send response to journal: Re: Re: Self-fulfilling Prophecy	Peter Singleton's observations are all very well as far as they go, but medicine is not always, nor, in its most culturally dominant forms, most often, a committee meeting. Multi-disciplinary teams and so on have their place, but the principal mode in which medicine is conducted involves a single patient consulting a single doctor alone and unobserved. The personality traits which are most appropriate for this encounter may not be those best fitted for a committee. Until scalpels can be wielded by committee and committees can sign prescriptions I suggest that we concentrate on finding those best fitted for personal medical and surgical practice. Competing interests: I was selected for medical school in the 1960's, twice.