Should we ditch impact factors?

doi:10.1136/bmj.39146.545752.BE

BMJ 2007;334:569 (17 March), doi:10.1136/bmj.39146.545752.BE

Feature

Should we ditch impact factors?

Richard Hobbs, head of primary care

Department of Primary Care and General Practice, University of Birmingham, Birmingham B15 2TT

f.d.r.hobbs{at}bham.ac.uk

Even advocates of impact factors admit that they are a flawedmeasure of quality. Gareth Williams believes we should get ridof them whereas Richard Hobbs thinks refinement is the answer

From 2008, state funding for academic research in the UK willbe calculated differently. The research assessment exercise,which is based substantially on (high intensity, high cost)peer review, will transfer to bibliometric scoring.¹ Such metricsinclude journal impact factors, published annually in JournalCitation Reports. The impact factor is based on the averagenumber of citations of individual "source" articles (originalpapers, research reports, and reviews) published in that journalduring the preceding two years.

Of course, as with most routinely collected data, there areproblems. Some of them are basic—only 2.5% of journalsare tracked² and not all disciplines routinely cite others'work. Numerous other non-quality factors drive article citationsand therefore the impact: reviews cite other articles the most³;basic sciences routinely cite many more references than clinicalsciences; short turnover science (principally basic) is widelyreferenced for a short time (enhancing scores in the limitedtwo year window); journals that publish weekly get more citationsthan monthly (probably because they are more widely read); specialistclinical journals are cited less than non-specialist; selectivecitation occurs, through within country bias (most notably inthe US),² within discipline bias,² and general gaming.

Despite these problems we need a measure of quality. We wantjournals to publish material that has been filtered to ensureit is reliable, interesting, relevant, or important and thatreading it results in some wider benefit. And surely, if weare sufficiently influenced by the work, we would want to acknowledgethe source through citation? Citation scores are therefore capableof recognising some value, some quality, of those articles thatpersuade us through argument or data.

So rather than just discarding impact factors we should considersolutions to the problems. For example, extend the citationsurveillance period beyond two years or adjust for citationbulge in rapid turnover disciplines. Weightings could be appliedto adjust for the average number of references across journals(which will be discipline dependent) or for national versusinternational citation; or we could consider scoring journalson only their most important papers, since the most cited 15%of articles account for 50% of citations and the most cited50% of articles account for 90% of the citations.⁴ Why not measureonly those that really influence rather than include the tail?After all, development of more complex citation scoring wasadvocated (to avoid citing unreliable studies and deepen historicalmeaning) by Eugene Garfield, the father of impact factors, inhis original 1955 paper.⁵

Fewer, better papers

An even more compelling reason to measure and publicise thequality of journals is that there are simply too many of them.An estimated 126 000 journals⁶ in a world that is strugglingto cope with waste is ridiculous. Who reads them all? How manyjournals exist merely to meet the inexorable drive to "publishat all costs," regardless of quality? Too much pressure on academicsto produce too many papers creates a market for journals, butat an intellectual, material, and environmental cost.

We should be attempting to encourage more complete work, ofa higher quality, in a reduced number of journals that are morewidely read and cited. It would be nice to think that a handfulof practice changing research papers or thought provoking reviewswere a worthwhile career objective rather than 200 papers ofaverage to low quality. Electronic publication of preliminarywork could stimulate further iterative study and critical responsewithout influencing citation scores. The journals with higherimpact factors currently produce expanded electronic versionsof papers that, if freely available, widen readership and influence.This system is also potentially more carbon friendly—weprint off fewer articles, probably only those 15% that accountfor most citations.

All of this is not to say that the articles representing thecitation tail in journals with some impact factor should bedisregarded. However, quality measures that focus on a reducednumber of papers in fewer journals should be encouraged.

So, yes, its easy to criticise bibliometrics, but we shouldattempt to refine them and debate in parallel how we can trackacademic careers and encourage fewer, but better studies thataffect the wider community. After all, as Oscar Wilde said,"The only thing to do with good advice is pass it on. It isnever any use to oneself," although he failed to reference PubliliusSyrus's maxim: "Many receive advice, few profit by it." So citefreely when citation is merited and let adjusted metrics profitthose, fewer, journals that meet a quality threshold. They mightalso help save the planet.

Competing interests: RH is a career academic dedicated to evidencebased care and regularly attempts to publish quality papersin quality journals and to get a high research assessment exercisescore (only one citation here).

References

Hobbs FDR, Stewart PM. The 2008 research assessment exercise: implications for UK medical research. BMJ 2006;332:983-94.[Free Full Text]
Taubes G. Measure for measure in science. Science 1993;260:884-6.[Free Full Text]
Joyce J, Rabe-Hesketh S, Wessely S. Reviewing the reviews. JAMA 1998;280:264-6.[Abstract/Free Full Text]
Seglan PO. The skewness of science. J Am Soc Information Sci 1992;43:628-38.[CrossRef]
Garfield E. Citation indexes for science: a new dimension in documentation through association of ideas. Science Int J Epidemiol 1955;122:108-11. (Reprinted
Seglan PO. Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314:497.[Free Full Text]