Rapid Responses to:
|
|
Rapid Responses: Rapid Responses published:
-
![[Read Rapid Response]](/icons/misc/sectionDOWN.gif)
Publications divided by research income to measure research efficacy
- Dietmar Fuchs (28 April 2006)
-
Simply counting number of publications may be the best research performance measure
- Bruce G Charlton, Peter Andras (30 April 2006)
-
OUTCOME OF RESEARCH
- Nazar R DESSOUKI (30 April 2006)
-
Rating research by scientific and clinical value
- Ellen C G Grant (2 May 2006)
-
Pedigree, paternity and progeny.
- Woody Caan (3 May 2006)
-
How should we rate conflict of interest?
- Trisha Greenhalgh (4 May 2006)
-
Academic Medicine, not all is by all means rosy
- Jangu Banatvala, Malcolm Symonds, Peter Bell (30 May 2006)
|
|
|||
|
Dietmar Fuchs, A.Univ.Prof. Innsbruck Medical University, 6020 Innsbruck, Austria
Send response to journal:
|
To simplify funding of academic research it is planned to introduce assessment based mainly on metrics as performance indicators. These may include research income, publications, citations, and numbers of research students. There seems to be only little doubt about the relevance of publications and citations scores to reflect scientific output. However, research income or number of students as such do not really reflect efficacy of research. Measures like publications and citations divided by research income and number of students could represent a much better judgement of the efficacy of a research group. Indeed, it will be no real help for the community when researchers are just able to achieve high research income but not an adequate output. Competing interests: None declared |
|||
|
|
|||
|
Bruce G Charlton, Editor-in-Chief, Medical Hypotheses University of Newcastle upon Tyne, NE1 7RU, UK, Peter Andras
Send response to journal:
|
There seems to be much confusion in the debate concerning how to replace the UK Research Assessment Exercise. For example, it should be noted that numbers of publications and citations are research output measures; while research council funding and numbers of academic and research staff are input measures. The big problem of the RAE is its non-transparency and lack of objectivity due to the unmeasurable role of subjective opinion - the same criticism applies to research council decisions. Therefore, it would be a mistake for future funding systems to try and replicate the results of the current RAE: we can do better. A system of research evaluation based on fundamental output metrics such as publications and/ or citations is (like any simple measure) inevitably incomplete and imperfect - but has the great advantage that these deficiencies are not concealed. A league table of English and Scottish university publications and citations accumulated in in the ISI Web of Science for 2000-4 gives plausible and consistent results. Publications (Citations) 1 (1) Cambridge 2 (2) Oxford 3 (4) Imperial College, London 4 (3) UCL 5 (6) Manchester 6 (5) Edinburgh 7 (7) Bristol 8 (8) Birmingham 9 (9) Glasgow 10 (10) Kings College, London 11 (12) Leeds 12 (11) Sheffield Since publication and citations rank order is so similar, this suggests that measurement of each university's annual publications may be exactly the simple, sensitive, objective and transparent research performance measure which is needed – and one that is superior to the complex, slow, subjective and methodologically obscure evaluations of the RAE. Competing interests: None declared |
|||
|
|
|||
|
Nazar R DESSOUKI, consultant surgeon NHS
Send response to journal:
|
The best way to rate research is the outcome of that research and the cost effect value. Competing interests: None declared |
|||
|
|
|||
|
Ellen C G Grant, physician and medical gynaecologist 20 Coombe Ridings, Kingston-upon-Thames, KT2 7JU, UK
Send response to journal:
|
Simply counting number of publications may be a measure of research performance but does the number of publications also relate to clinical relevance and importance? A search under zinc deficiency at www.pubmed.gov brings up 6420 publications. It is inexplicable that the great clinical importance of accurately diagnosing and repleting zinc deficiency for normal immune function has not yet been recognised by most practising clinical biochemists or clinicians. In contrast, there are 16224 references to hormone replacement therapy although use of progestins and oestrogens has been proved to increase the risk of breast and endometrial cancers, strokes, thromboses and myocardial infarctions. The number of publications may reflect funding motivation rather than genuine long-lasting scientific merit. Competing interests: None declared |
|||
|
|
|||
|
Woody Caan, Professor of public health Anglia Ruskin University, Chelmsford CM1 1SQ, UK.
Send response to journal:
|
What do we really mean by assessing the ‘quality’ of health research? [1] We can give the editors of the Lancet and New England Journal of Medicine the power to shape UK research funding for the coming decade (because wide circulation generic publications produce the biggest impact in the US-focused Web of Knowledge) if we base funding on past citations. But is the quality of a ‘fashionable’ author who published on the molecular biology of heart disease inherently ‘better’ than an ear nose and throat surgeon who published in a smaller, specialist field like neuroma? We can concentrate future funding even more on high-cost (and high-risk) projects, but is the quality of research using expensive isotopes and scanners inherently ‘better’ than research in primary care on breastfeeding? The Higher Education Funding Council (HEFCE) has no interest in improving the clinical care in the National Health Service nor in the public well-being of society as a whole. [2] HEFCE will always go for the concentration of money by the most direct way into their favoured institutions (consider the way the results of the 2001 Research Assessment Exercise were fiddled, post-hoc) because it makes their life simple to deal with a few big players rather than stimulate innovative or world- changing research. Metrics focused on past citations and grants will ensure this cosy concentration on the same old friends of HEFCE: the only future change in the game might be higher education mergers (as in Manchester) into even bigger players. No wonder the Greater Manchester Research Alliance pioneered the ‘Research Passport’! The far-sighted director of the Economic and Social Research Council (Ian Diamond) gave the Academy for the Social Sciences a fascinating briefing on metrics in 2005: use of existing citation databases like the US Web of Knowledge profoundly understates the strengths of small disciplines, especially when published materials originate in the European Union. A key concept to which the ESRC introduced me was ‘esteem’, by which they meant something dynamic and cumulative (not just on how many public committees or inquiries someone sat). Subsequently I have spent a year pondering what qualities are admirable or judged excellent in scientific esteem, and so far have identified two dimensions that could be measured. Both proposals stemmed from encountering the health scientist that I had most ‘esteemed’: Peter Medawar, although they were also readily detectable in meeting other Nobel laureates who had worked in the UK, such as Nikolaas Tinbergen, Edgar Adrian, John Eccles or Frederick Sanger. One is a pedigree of ideas (seeking the ‘origin’ in originality), and one is a pedigree of influence (‘shaping’ the shapers). What are the seminal ideas that gave paternity to a widely growing family tree of research (like Medawar’s immunology)? This is not at all the same as the most highly cited papers, which may actually be derivative and technical. It needs a mapping exercise (sometimes called data mining in areas like patents) to identify the seminal work, and in many areas of health these family trees are not hard to map (think, say, worldwide of diagnostic magnetic resonance and the trees are only 30 years old with perhaps three original seeds?). With apologies for my sexist language, but I imagine Dorothy Hodgkin or Marie Curie showed a similar pattern of ‘paternity’ in their disciplines. One common feature of all the laureates, above, is that they nurtured teams and networks, from which new research leaders emerged. The Fields Medal is the mathematics equivalent of the Nobel prise for medicine. Mathematicians seem more aware that working with one paradigm-shifter can help shape other trail-blazers. In the early decades of the Institute of Psychiatry, Aubrey Lewis had an extraordinary influence on promoting academic careers and charting the challenges for mental health research. [3] This influence lasted at least 60 years (while at the Maudsley Hospital in the 1990s one could still trace Lewis’ progeny) and spanned the British Commonwealth. The fecundity of Lewis can be measured by the amazing quantity and variety of research still produced by his one small Institute, but even more so by measuring the international collaborations that have nurtured research leaders far beyond the UK. Now that is a legacy of ‘quality’. 1 Hobbs FDR, Stewart PM. How should we rate research? BMJ 2006; 332: 983-984. 2 Caan W. Inequalities and research need to be balanced. BMJ 2002; 324: 51-52. 3 Angel K, Jones E, Neve M. (eds), European psychiatry on the eve of war: Aubrey Lewis, the Maudsley Hospital and the Rockefeller Foundation in the 1930s. Medical History, Supplement No 22, London: Wellcome Trust Centre for the History of Medicine at University College London, 2003. [With many apologies for such a long 'letter' - this has become more like a 'personal opinion'.] Competing interests: Once, briefly, held an honorary lectureship at the IoP. Was once a member of the Foresight forum of the AcSS. |
|||
|
|
|||
|
Trisha Greenhalgh, Professor of Primary Health Care University College London
Send response to journal:
|
What on earth possessed the BMJ to commission an editorial on the RAE from two people who form part of the machinery for implementing and perpetuating it? This editorial has led to widespread dissent in the corridors of university departments. But since these people are to be the ones marking our homework, why would we want to upset them by disagreeing with them in print? Richard Smith once wrote an editorial on death and declared as a conflict of interest "I was recently very upset by the death of my pet rabbit". I'm confident that, if she put her mind (and perhaps a working group) to it, Fiona Godlee could take the BMJ's conflict of interest policy to a higher level of sophistication. Competing interests: Will be submitting to the RAE as part of the UCL return |
|||
|
|
|||
|
Jangu Banatvala, Emeritus Professor of Clinical Virology, King's College London Home CM22 6AN, Malcolm Symonds, Peter Bell
Send response to journal:
|
Editor: In their article on “How should we rate research” Hobbs & Stewart present a rather rosy view of what has been achieved by the Research Assessment Exercise (RAE) and are encouraged by the Government’s intention to simplify the procedure(1). This is based essentially on a metrics based assessment, rather than the peer reviewed exercise. The review of the literature in the Editorial was somewhat selective, e.g. some papers (2) and letters (3) expressing a contrary view were not cited, even if only to criticise their viewpoints. It is surprising that there was scarcely any mention of teaching. It is encouraging, however, that the MRC and the DH’s R&D will now hold a single fund for health related research and this should do more to encourage opportunities for translational research. Its success will be dependent on collaboration between Heads of Medical Schools and NHS Chief Executives, not always known for their collaborative approach, as well as such Research Institutes as the MRC. The new initiative must be protected from being absorbed by making good budgetary deficits, any Government’s proposals to set unrealistic targets for NHS Trusts and NHS research, which is not of the highest quality. The major question, however, which needs to be addressed relates to the recruitment and retention of clinical academic staff. Despite a 40% increase in Medical School intake since 2000, there were 84 vacant Chairs in medical specialities by 2005(4). Between 2002 and 2004, clinical lecture grades were reduced by 42% thereby removing the seed corn of future academic posts. Pathology has experienced a marked reduction and worryingly, in some schools, this discipline has only a marginal role in medical undergraduate education. Consequently, those now qualifying may have little understanding of the mechanisms by which disease is produced and which clinical laboratory investigations should be selected for diagnosis. Particularly in such craft specialities as surgery, and obstetrics and gynaecology lecturers no longer view academic medicine as an attractive career option, despite opportunities for research in such areas as vascular, including coronary artery, surgery, endoscopic techniques and joint replacement. Research and teaching assessments, clinical appraisals by the GMC and locally, as well as clinical responsibilities deter the young from an academic career. The BMJ Editorial has not provided any reassurance that the new system will address the blight, much of which must be laid at the feet of the RAE (2). The way forward must be to give every encouragement for ensuring that high standards of basic medical sciences, clinical practice and teaching are supported in close proximity. Contrary to views sometimes expressed, basic scientists have an important role in teaching and not only for intercalated BScs. Teams who assess by the newly established criteria must have vision and experience and take into account clinical practice, research and teaching, giving appropriate weight to each, as must be the case for Schools of Medicine. A previous review conducted by the Wessex Institute concluded that, in medicine, compared with other disciplines, assessors judged their peers unduly harshly. “Quis custodiet ipsos custodies?” Juvenal. Vi.347-8. (1) Hobbs F.D. & Stewart P.M. How should we rate research? BMJ 332:983-984 (2) Banatvala J. Bell P. Symonds M. The Research Assessment Exercise is bad for UK Medicine. The Lancet 2005 365:458-460 (3) Symonds E.M. Bell P. Banatvala J. The Future of Academic Medicine looks bleak. BMJ 351:694 (4) Clinical Academic Staffing Levels in UK Medical and Dental Schools: Data update 2004 Council of Heads of Medical Schools June 2005 Professor J.E. Banatvala CBE, Emeritus Professor of Clinical Virology, University of London Professor Sir Peter Bell, Emeritus Professor of Surgery, University of Leicester Professor E.M. Symonds, Emeritus Professor of Obstetrics & Gynaecology, University of Nottingham Competing interests: None declared |
|||