Rapid Responses to:

LETTERS: K S Joseph Quality of impact factors of general medical journals BMJ 2003; 326: 283 [Full text]

Rapid Responses: Submit a response to this article

Rapid Responses published:

Quality matters --and the choice of indicator, too

Miquel Porta   (6 February 2003)

Quality of impact factors of general medical journals - PRAVDA wins hands down

Tom Jefferson, Karen Shashok   (8 February 2003)

Re: Quality of impact factors of general medical journals - PRAVDA wins hands down

Eugene Garfield   (19 February 2003)

The assessment of research quality using a combination of approaches

Joseph L.Y. Liu   (19 February 2003)

Quality matters --and the choice of indicator, too 6 February 2003
Miquel Porta, Institut Municipal d'Investigació Mèdica (IMIM), and Universitat Autònoma de Barcelona E-08003 Barcelona, Spain Send response to journal: Re: Quality matters --and the choice of indicator, too	Results of the thoughtful analysis published by K S Joseph add new data --and a remarkable twist-- to existing knowledge on weaknesses in the accuracy of the data that ISI (now part of Thomson) [1] extracts from journals. Though ISI has long struggled to avoid mistakes, the vast amount of data needed to produce their products stresses the importance of quality checks. Usually, these controls are impossible to perform by users, most of whom do not have access to the original, raw data --for instance, on which articles were counted to be part of the denominator of the bibliographic "impact factor" (BIF) [2]. The results offered by Joseph are also a reminder --long emphasised by Eugene Garfield, almost from his first writings 50 years ago [3-13]-- that the BIF is often not the scientometric indicator of choice. Specifically, if you wish to know the "impact" of a journal (if the journal is really your unit of analysis and interest), then you should first consider looking at the total number of citations received by such journal [2]. In passing you are very likely to avoid the pitfall uncovered by Joseph: the total number of citations received is not much influenced by the "number of items" chosen to compute the BIF. With increasing access via the www to ISI's data, more attention is being devoted to the specific number of citations received by each individual article. This should further contribute to avoid another intrinsic "weakness" of BIF, for which no one is to blame: BIF is just the ("misleading") average of a highly skewed distribution (often, 85% of citations received "by a journal" are actually received by about 15% of the articles it published) [2,14,15]. I'm afraid that much of BIF's "sex-appeal" stems precisely from the fact that an average is so simple a measure. But as scientists we surely can go beyond... References 1. http://www.thomson.com and http://www.isinet.com/isi/ [last accessed on Febr 6, 2003]. 2. Porta M. The bibliographic “impact factor” of the Institute for Scientific Information, Inc.: how relevant is it really for public health journals? J Epidemiol Community Health 1996; 50: 606-610. 3. Garfield E. Citation analysis as a tool in journal evaluation. Science 1972; 178: 471-479. 4. Garfield E. Significant journals of science. Nature 1976; 264: 609 -615. 5. Garfield E. Caution urged in use of citation analyses. Trends Biochem Sci 1977; 2: 84. 6. Garfield E. Is citation analysis a legitimate evaluation tool?. Scientometrics 1979; 1 (4): 359-375. 7. Garfield E. How to use citation analysis for faculty evaluations and when is it relevant?. Part 1. Current Contents Life Sciences 1983; 44: 5-13. Part 2. Current Contents Life Sciences 1983; 45: 5-14. 8. Garfield E. Uses and misuses of citation frequency. Current Contents 1985; 43: 3-9. 9. Garfield E. Which medical journals have the greatest impact?. Ann Intern Med 1986; 10: 313-320. 10. Garfield E. The 101 most-cited papers from the British Medical Journal highlight the important role of epidemiology in medicine. Current Contents 1987; 7 (Febr): 3-11. 11. Garfield E. Prestige versus impact: Established images of journals, like institutions, are resistant to change. Current Contents Life Sciences 1987; 38 (30): 3-4. 12. Garfield E. Which Medical Journals Have the Greatest Impact? Ann Internal Med 1986; 105: 313-320. 13. Most of these articles have been compiled in the volumes of Essays of An Information Scientist, see: http://www.garfield.library.upenn.edu/ [last accessed on Febr 6, 2003]. 14. Seglen PO. Citation frequency and journal impact: Valid indicators of scientific quality? J Intern Med 1991; 229: 109-111. 15. Seglen PO. How representative is the journal impact factor? Research Evaluation 1992; 2: 143-149. Competing interests:   None declared
Quality of impact factors of general medical journals - PRAVDA wins hands down 8 February 2003
Tom Jefferson, Director Health Reviews Ltd, Karen Shashok Send response to journal: Re: Quality of impact factors of general medical journals - PRAVDA wins hands down	EDITOR - Joseph implies international recognition that citations are a hallmark of quality of scientific articles and their host journals. We must congratulate Joseph1, as he has successfully re-introduced a concept which had last been aired during the cold war: namely that Pravda was the best daily in the world as it was read by over 30 million people. Citations and impact factors are to do with circulation, not with quality. They belong to the world of publishing, not that of science. Although widely cited journals are read more often than others, we have no evidence that what they publish is of higher quality. Indeed, there is no internationally accepted definition of quality2. While interesting, Joseph's examination of how changes in impact factors were related to changes in the number of citable items in leading journals does not add support to the argument that the impact factor is an accurate indicator of the quality of research articles—quite the contrary, in fact. Did the quality of articles in JAMA or the Lancet change together with the rises or falls in their IF? Lastly, far from assessing the quality of scientific production "which underscores the altruistic nature of the scientific enterprise", citation rates are being used to apportion research money, and may potentially tie directly into researcher's pockets3. Thus a circulation business indicator is being used to affect healthcare. Tom Jefferson Via Adige 28a,00061 Anguillara Sabazia,Rome,Italy email toj1@aol.com Karen Shashok C/ Comp. Ruiz Aznar 12, 2-A.18008 Granada,Spain email kashashok@wanadoo.es 1. Joseph KS. Quality of impact factors of general medical journals. BMJ 2003;326:283 ( 1 February ) 2. Jefferson TO, Wager E, Davidoff F. Measuring the quality of editorial peer review. JAMA 2002; 287:2786-90 3. Jimenez-Contreras E, Delgado-Lopez-Cozar E, Ruiz-Perez R, Fernandez VM. Impact-factor rewards affect Spanish research. Nature 2002; 417:898 Competing interests:   Tom Jefferson co-authored the books "Peer-review in Health Sciences" and "How to Survive Peer Review". Both authors are active peer-reviewers and have published articles in peer-reviewed journals.
Re: Quality of impact factors of general medical journals - PRAVDA wins hands down 19 February 2003
Eugene Garfield, Founding Editor The Scientist Send response to journal: Re: Re: Quality of impact factors of general medical journals - PRAVDA wins hands down	If circulation were the determining factor in journal impact then JAMA should have the highest impact factor and journals like NEJM with lower circulation would not. Impact is primarily a measure of the use (value?) by the research community of the article in question.If an author or journal is cited significantly above the average then we say the author's work has been influential albeit sometimes controversial. It is true that quality like beauty is often in the eyes of the beholder, but if peer judgments are taken as a potential source of quality judgment then citation frequency is well correlated with e.g. Nobel and other awards. It is extremely rate for a Nobel class scientist not to have published one or more citation classics. Indeed in 1967 we determed that Nobel scientists publish five to six times as often as the average author and their work is citd 30 to fifty times as often. There is extensive documentation at www.eugenegarfield.org Competing interests:   None declared
The assessment of research quality using a combination of approaches 19 February 2003
Joseph L.Y. Liu, Research Fellow Centre for Statistics in Medicine, Institute of Health Sciences, Oxford University, Oxford OX3 7LF Send response to journal: Re: The assessment of research quality using a combination of approaches	Although Porta's suggestion on the use of the number of citations received by each article is an improvement over the journal impact factor as a measure of a publication's research quality [1][2], it does not solve the problem of different practices in citing references between disciplines, which is largely unrelated to quality [3]. For example, when a title search was conducted for articles in 1995 on the ISI Web of Science database (accessed on February 10 2003), I found that the most highly cited papers in the BMJ, Lancet, New England Journal of Medicine, JAMA and Annals of Internal Medicine on malaria (the top ten papers in this area were cited an average of 64 times) and diarrhoea (31 times) are substantially lower than those for coronary heart disease (435 times) and breast cancer (289 times). As I had suggested previously as a co-author of a letter on impact factors [3], the citation number of individual papers should be adjusted according to discipline to improve on an imperfect but widely used indicator of research quality. However, the inherent limitations of a single numerical summary measure [4] and the lack of empirical evidence on the effectiveness of traditional peer review [5] indicate the need to assess research quality using a combination of approaches, including post publication peer review [6], indicators of the social impact of research [7], and the evaluation of research performance by independent expert panels using transparent and evidence based criteria. [1] Joseph KS. Quality of impact factors of general medical journals. BMJ 2003; 326: 283. [2] Porta M. Quality matters-and the choice of indicator, too. Available from URL: http://bmj.com/chi/eletters/326/7383/283. [3] Tang JL, Wong TW, Liu JLY. Adjusted impact factors for comparisons between disciplines. J Epidemiol Community Health 1999[letter]; 53: 739- 40. [4] Smith R. Unscientific practice flourishes in science. BMJ 1998; 317: 1036-1040. [5] Jefferson TO, Alderson P, Davidoff F, Wager E. Editorial peer-review for improving the quality of reports of biomedical studies (Cochrane Methodology Review). In: The Cochrane Library, Issue 1, 2003. Oxford: Update Software. [6] Altman DG. Poor-quality medical research. What can journals do? JAMA 2002; 287: 2765-2767. [7] Smith R. Measuring the social impact of research: difficult but necessary. BMJ 2001; 323: 528. Competing interests:   None declared