What do we really mean by assessing the ‘quality’ of health research?
[1] We can give the editors of the Lancet and New England Journal of
Medicine the power to shape UK research funding for the coming decade
(because wide circulation generic publications produce the biggest impact
in the US-focused Web of Knowledge) if we base funding on past citations.
But is the quality of a ‘fashionable’ author who published on the
molecular biology of heart disease inherently ‘better’ than an ear nose
and throat surgeon who published in a smaller, specialist field like
neuroma? We can concentrate future funding even more on high-cost (and
high-risk) projects, but is the quality of research using expensive
isotopes and scanners inherently ‘better’ than research in primary care on
breastfeeding? The Higher Education Funding Council (HEFCE) has no
interest in improving the clinical care in the National Health Service nor
in the public well-being of society as a whole. [2] HEFCE will always go
for the concentration of money by the most direct way into their favoured
institutions (consider the way the results of the 2001 Research Assessment
Exercise were fiddled, post-hoc) because it makes their life simple to
deal with a few big players rather than stimulate innovative or world-
changing research. Metrics focused on past citations and grants will
ensure this cosy concentration on the same old friends of HEFCE: the only
future change in the game might be higher education mergers (as in
Manchester) into even bigger players. No wonder the Greater Manchester
Research Alliance pioneered the ‘Research Passport’!
The far-sighted director of the Economic and Social Research Council
(Ian Diamond) gave the Academy for the Social Sciences a fascinating
briefing on metrics in 2005: use of existing citation databases like the
US Web of Knowledge profoundly understates the strengths of small
disciplines, especially when published materials originate in the European
Union. A key concept to which the ESRC introduced me was ‘esteem’, by
which they meant something dynamic and cumulative (not just on how many
public committees or inquiries someone sat). Subsequently I have spent a
year pondering what qualities are admirable or judged excellent in
scientific esteem, and so far have identified two dimensions that could be
measured. Both proposals stemmed from encountering the health scientist
that I had most ‘esteemed’: Peter Medawar, although they were also readily
detectable in meeting other Nobel laureates who had worked in the UK, such
as Nikolaas Tinbergen, Edgar Adrian, John Eccles or Frederick Sanger.
One is a pedigree of ideas (seeking the ‘origin’ in originality), and one
is a pedigree of influence (‘shaping’ the shapers).
What are the seminal ideas that gave paternity to a widely growing
family tree of research (like Medawar’s immunology)? This is not at all
the same as the most highly cited papers, which may actually be derivative
and technical. It needs a mapping exercise (sometimes called data mining
in areas like patents) to identify the seminal work, and in many areas of
health these family trees are not hard to map (think, say, worldwide of
diagnostic magnetic resonance and the trees are only 30 years old with
perhaps three original seeds?). With apologies for my sexist language,
but I imagine Dorothy Hodgkin or Marie Curie showed a similar pattern of
‘paternity’ in their disciplines.
One common feature of all the laureates, above, is that they nurtured
teams and networks, from which new research leaders emerged. The Fields
Medal is the mathematics equivalent of the Nobel prise for medicine.
Mathematicians seem more aware that working with one paradigm-shifter can
help shape other trail-blazers. In the early decades of the Institute of
Psychiatry, Aubrey Lewis had an extraordinary influence on promoting
academic careers and charting the challenges for mental health research.
[3] This influence lasted at least 60 years (while at the Maudsley
Hospital in the 1990s one could still trace Lewis’ progeny) and spanned
the British Commonwealth. The fecundity of Lewis can be measured by the
amazing quantity and variety of research still produced by his one small
Institute, but even more so by measuring the international collaborations
that have nurtured research leaders far beyond the UK. Now that is a
legacy of ‘quality’.
1 Hobbs FDR, Stewart PM. How should we rate research? BMJ 2006; 332:
983-984.
2 Caan W. Inequalities and research need to be balanced. BMJ 2002;
324: 51-52.
3 Angel K, Jones E, Neve M. (eds), European psychiatry on the eve of
war: Aubrey Lewis, the Maudsley Hospital and the Rockefeller Foundation in
the 1930s. Medical History, Supplement No 22, London: Wellcome Trust
Centre for the History of Medicine at University College London, 2003.
[With many apologies for such a long 'letter' - this has become more
like a 'personal opinion'.]
Competing interests:
Once, briefly, held an honorary lectureship at the IoP. Was once a member of the Foresight forum of the AcSS.
Rapid Response:
Pedigree, paternity and progeny.
What do we really mean by assessing the ‘quality’ of health research?
[1] We can give the editors of the Lancet and New England Journal of
Medicine the power to shape UK research funding for the coming decade
(because wide circulation generic publications produce the biggest impact
in the US-focused Web of Knowledge) if we base funding on past citations.
But is the quality of a ‘fashionable’ author who published on the
molecular biology of heart disease inherently ‘better’ than an ear nose
and throat surgeon who published in a smaller, specialist field like
neuroma? We can concentrate future funding even more on high-cost (and
high-risk) projects, but is the quality of research using expensive
isotopes and scanners inherently ‘better’ than research in primary care on
breastfeeding? The Higher Education Funding Council (HEFCE) has no
interest in improving the clinical care in the National Health Service nor
in the public well-being of society as a whole. [2] HEFCE will always go
for the concentration of money by the most direct way into their favoured
institutions (consider the way the results of the 2001 Research Assessment
Exercise were fiddled, post-hoc) because it makes their life simple to
deal with a few big players rather than stimulate innovative or world-
changing research. Metrics focused on past citations and grants will
ensure this cosy concentration on the same old friends of HEFCE: the only
future change in the game might be higher education mergers (as in
Manchester) into even bigger players. No wonder the Greater Manchester
Research Alliance pioneered the ‘Research Passport’!
The far-sighted director of the Economic and Social Research Council
(Ian Diamond) gave the Academy for the Social Sciences a fascinating
briefing on metrics in 2005: use of existing citation databases like the
US Web of Knowledge profoundly understates the strengths of small
disciplines, especially when published materials originate in the European
Union. A key concept to which the ESRC introduced me was ‘esteem’, by
which they meant something dynamic and cumulative (not just on how many
public committees or inquiries someone sat). Subsequently I have spent a
year pondering what qualities are admirable or judged excellent in
scientific esteem, and so far have identified two dimensions that could be
measured. Both proposals stemmed from encountering the health scientist
that I had most ‘esteemed’: Peter Medawar, although they were also readily
detectable in meeting other Nobel laureates who had worked in the UK, such
as Nikolaas Tinbergen, Edgar Adrian, John Eccles or Frederick Sanger.
One is a pedigree of ideas (seeking the ‘origin’ in originality), and one
is a pedigree of influence (‘shaping’ the shapers).
What are the seminal ideas that gave paternity to a widely growing
family tree of research (like Medawar’s immunology)? This is not at all
the same as the most highly cited papers, which may actually be derivative
and technical. It needs a mapping exercise (sometimes called data mining
in areas like patents) to identify the seminal work, and in many areas of
health these family trees are not hard to map (think, say, worldwide of
diagnostic magnetic resonance and the trees are only 30 years old with
perhaps three original seeds?). With apologies for my sexist language,
but I imagine Dorothy Hodgkin or Marie Curie showed a similar pattern of
‘paternity’ in their disciplines.
One common feature of all the laureates, above, is that they nurtured
teams and networks, from which new research leaders emerged. The Fields
Medal is the mathematics equivalent of the Nobel prise for medicine.
Mathematicians seem more aware that working with one paradigm-shifter can
help shape other trail-blazers. In the early decades of the Institute of
Psychiatry, Aubrey Lewis had an extraordinary influence on promoting
academic careers and charting the challenges for mental health research.
[3] This influence lasted at least 60 years (while at the Maudsley
Hospital in the 1990s one could still trace Lewis’ progeny) and spanned
the British Commonwealth. The fecundity of Lewis can be measured by the
amazing quantity and variety of research still produced by his one small
Institute, but even more so by measuring the international collaborations
that have nurtured research leaders far beyond the UK. Now that is a
legacy of ‘quality’.
1 Hobbs FDR, Stewart PM. How should we rate research? BMJ 2006; 332:
983-984.
2 Caan W. Inequalities and research need to be balanced. BMJ 2002;
324: 51-52.
3 Angel K, Jones E, Neve M. (eds), European psychiatry on the eve of
war: Aubrey Lewis, the Maudsley Hospital and the Rockefeller Foundation in
the 1930s. Medical History, Supplement No 22, London: Wellcome Trust
Centre for the History of Medicine at University College London, 2003.
[With many apologies for such a long 'letter' - this has become more
like a 'personal opinion'.]
Competing interests:
Once, briefly, held an honorary lectureship at the IoP. Was once a member of the Foresight forum of the AcSS.
Competing interests: No competing interests