Education and debate

Fortnightly Review: How can impact factors be improved?

^a Institute for Scientific Information, 3600 Market Street, Suite 450, Philadelphia, PA 19104, USA

Impact factors are widely used to rank and evaluate journals. They are also often used inappropriately as surrogates in evaluation exercises. The inventor of the Science Citation Index warns against the indiscriminate use of these data. Fourteen year cumulative impact data for 10 leading medical journals provide a quantitative indicator of their long term influence. In the final analysis, impact simply reflects the ability of journals and editors to attract the best papers available.

Counting references to rank the use of scientific journals was reported as early as 1927 by Gross and Gross.¹ In 1955 I suggested that reference counting could measure "impact,"² but the term "impact factor" was not used until the publication of the 1961 Science Citation Index (SCI) in 1963. This led to a byproduct, Journal Citation Reports (JCR), and a burgeoning literature using bibliometric measures. From 1975 to 1989, JCR appeared as supplementary volumes in the annual SCI. From 1990-4, they have appeared in microfiche, and in 1995 a CD ROM edition was launched.

View larger version (111K): [in this window] [in a new window]   Large journals that publish many papers may not have as high an impact as smaller review journals

Calculation of current impact factors

The most used data in the JCR are impact factors--ratios obtained from dividing citations received in one year by papers published in the two previous years. Thus, the 1995 impact factor counts the citations in 1995 journal issues to "items" published in 1993 and 1994. I say "items" advisedly. There are a dozen major categories of editorial matter. JCR's impact calculations are based on original research and review articles, as well as notes. Letters of the type published in the BMJ and the Lancet are not included in the publication count. The vast majority of research journals do not have such extensive correspondence sections. The effects of these differences in calculating journal impact can be considerable.^{3 4}

Absolute citation counts

The ubiquitous and sometimes misplaced use of journal impact factors for evaluation has caused considerable controversy. They are probably the most widely used of all citation based measures. They were invented to permit reasonable comparison between large and small journals. Absolute citation counts preferentially give highest rank to the largest or the oldest journals. For example, in 1994, articles published in the BMJ regardless of age were cited 37 600 times. Of these, 5800 citations--about 15%--were to items published in 1992 and 1993.

Table 1 shows absolute citation counts for nine English language clinical journals as well as the Journal of Biological Chemistry, which is included to emphasise the difference between absolute and relative citation of large journals. Table 2 lists these same journals ranked by 1994 impact. It is important to note that of the thousands of journals published and cited in SCI only 337 achieved a current impact higher than 3.0. The SCI processes over 3300 source journals and cites thousands more, but it does not include as sources hundreds of low impact applied and clinical journals. This has been a source of frustration to editors from the Third World. They often ask how they can improve impact so as to warrant inclusion.

Table 1--Absolute citation counts, 1994 and 1989
-------------------------------------------------------------------------------------
                                            Times cited                Times cited
                                            (thousands)                (thousands)
                                 Rank in      in 1994       Rank in      in 1989
Title                             1994    (from all years)   1989    (from all years)
-------------------------------------------------------------------------------------
Journal of Biological Chemistry      1         265.3            1         183.4
New England Journal of Medicine     10          98.9            9          72.4
Lancet                              11          85.5           12          68.0
JAMA                                41          41.0           40          29.4
BMJ                                 45          37.7           34          32.6
Annals of Internal Medicine         51          31.7           48          26.5
American Journal of Medicine        75          22.1           60          21.7
Archives of Internal Medicine      138          13.7          130          10.4
Medicine                           502           4.3          421           3.9
Annual Review of Medicine         1118           1.6          999           1.4

Table 2--1994 current impact factors
-----------------------------------------------------------------
                                   1994 impact       Citations to
                                      factor        1992-3 items
Journal                          (1992-3 articles)  (thousands)
-----------------------------------------------------------------
New England Journal of
  Medicine                             22.7             18.4
Lancet                                 17.3             16.6
Annals of Internal Medicine             9.9              5.4
Journal of Biological Chemistry         7.7             60.2
JAMA                                    6.9              8.1
BMJ                                     4.4              5.8
Archives of Internal Medicine           4.1              2.1
Medicine                                3.9              0.2
Annual Review of Medicine               2.8              0.2
American Journal of Medicine            2.7              1.7

Impact of review journals

It should be apparent from reviewing the two tables that the ranking relationship between quality and citation is not absolute. The Journal of Biological Chemistry is one of the most cited journals in the history of science but it is also among the largest. Publishing over 4000 research manuscripts a year (as do other high volume journals like Physical Review) inevitably leads to considerable variation in quality and impact of individual articles. As a consequence, while these journals publish many papers which become "citation classics," their current impact may not be as high as some smaller journals, especially certain review journals. Indeed, one of the highest impact journals is Annual Review of Biochemistry--with a current impact of 42.2. But as the data for Annual Review of Medicine show, this is not an absolute rule. In general, a journal is well advised to publish authoritative review articles if it would increase its impact. Since there are over 40 000 review articles published each year, not all will achieve high impact. Again, selection of reviews about active research fronts is important, as is their timing. Controversial topics may increase impact. A non-medical example is "cold fusion" by Fleischman and Pons,⁵ which has been cited over 500 times. A recent medical example could be the "Concorde study," already cited in 150 papers.⁶ But hundreds of halfbaked controversial ideas are essentially ignored.

Nothing will replace the judgment necessary for editors to select putative citation classics and to reject trivial or outlandish papers. Nevertheless, most reputable journals have at one time or another rejected papers which proved to be blockbusters. In retrospect, we should congratulate editors who publish controversial ideas such as that of Barry Marshall concerning Heliobacter pylori and peptic ulcers.⁷ That paper was well on its way to citation classic status when I nominated Marshall for the John Scott Award of Philadelphia--several years before he received the Albert Lasker award this year.

Method

papers

It is widely believed that method papers are cited more than the average and thus increase journal impact. How lovely it would be if every method proved to be another Lowry method,⁸ cited over 8000 times in 1994 and over 250 000 in its lifetime. But the fact is that method journals do not achieve extraordinary impact since the vast majority of their papers, like clinical tests, are not unusual.

An editor could select authors on the basis of past performance. By checking their citation histories, one could undoubtedly increase the probability of publishing papers with higher potential impact. Some editors do this instinctively, especially when publishing the first few issues of a new specialty journal. In most cases, the most-cited papers for newer journals appear in the first volume published.

Over the years, the increase in multiauthored papers has been apparent. This is matched by an increase in multinational and multi-institutional clinical and epidemiological studies. At the Institute for Scientific Information (ISI), unpublished studies support the notion that these papers produce greater impact. ISI's Science Watch has regularly reported on the most-cited current papers in medicine. There is fierce competition among editors to publish these "hot papers." These undoubtedly contribute to increased current impact. But what about long term impact?

Cumulative impact factors

In table 3, the 14 year citation impact for 1981-94 for the articles published from 1981 to 1986 in the BMJ or the Lancet are, in general, even higher than suggested by current impact. Annals of Internal Medicine is even stronger on this measure. Certain fields or topics require more time to mature because of delayed recognition (as with H pylori) or because of the time required to produce experimental or clinical results. It generally takes more time to achieve impact in dermatology than it does in molecular biology or astrophysics. Invidious comparisons between journals even in different fields of medicine do not take these subtleties into account.

Table 3--Cumulative impact 1981-94 for papers
published from 1981 to 1986
-----------------------------------------------------------
Title                            Impact   Papers  Citations
-----------------------------------------------------------
New England Journal of
  Medicine                        115.7    2 289   264 768
Annals of Internal Medicine        67.9    1 741   100 770
Medicine                           56.5      193    10 903
Journal of Biological Chemistry    50.8   14 483   735 026
Lancet                             50.5    3 544   179 125
American Journal of Medicine       28.2    2 932    82 649
Annual Review of Medicine          25.3      260     6 315
JAMA                               23.9    3 326    79 339
BMJ                                17.4    5 748   100 154
Archives of Internal Medicine      16.8    2 027    34 078

No substitute for judgment

Successful editors and publishers know that in order to improve the editorial quality of journals, there is no substitute for judgment, quality, and relevance. Impact and other citation measures merely report the facts. Authors will gravitate to journals with widespread influence. Circulation alone will not increase research impact. Otherwise, JAMA and others would rank higher in impact. But only a fraction of doctors do research. Dissemination of research results to the press may increase general awareness.⁹ But that role in research is played primarily by current awareness services such as Current Contents, by contacts at meetings, reprint exchange, and reading primary journals. Last but not least, the way to improve impact is to insist that authors cite all of the relevant literature. Editors should avoid artificial limits on bibliography as long as it is not obviously self serving.

In spite of dozens of presentations by myself and others, there continues to be a certain mystique about journal selection for Current Contents, the Science Citation Index, and the Social Sciences Citation Index. My 1990 essay in Current Contents is still sent to those making such enquiries.¹⁰

Of the 4500 journals covered by SCI and SSCI, probably 3000 can be described as biomedically related. Of these, 500 account for 50% of what is published and 75% of what is cited. Of the 3300 covered in Medline, hundreds of low impact journals are not included by ISI for similar reasons--space and economics.

Other factors in journal evaluation

New journals continue to appear each year and must be evaluated as early as possible. But even Nature in its periodic reviews of new journals requires the passage of time before accepting journals to be reviewed. An experienced evaluator takes into account timing, format, subject matter, past performance, and other indicators such as internationality. The first issue of many journals is full of hope, but they soon exhaust the backlog of material needed to insure continued, timely publication. Clearly a society publisher with years of experience does not launch a journal without a long term commitment--and its editorial standards will be well known. Inexperienced publishers often do not live up to their rosy expectations. The inclusion of abstracts and complete author, street, and email addresses are but a few artefacts that are factored in the judgment of minimum quality. ISI may also ask how well a particular specialty or country is represented in its coverage. And, after a few years of history, all other criteria being equal, one can look at a journal's impact.

This article has focused on journal impact factors and their role in what Stephen Lock described as "journalology."¹¹ Like individual authors, a variety of indicators can be used to judge journals in a current or historical sense. Impact numbers are probably less important than the rankings which are obtained. Often there are only slight quantitative differences involved. The literature is replete with recommendations for corrective factors that should be considered, but in the final analysis subjective peer judgment is essential.

Caution in use of impacts as surrogates

Journal impact data have been grafted on to certain large scale studies of university departments and even individuals. Sometimes a journal's impact is used as a substitute for the evaluation of recently published articles simply because it takes several years for the average article to be cited. However, a small percentage of articles will experience almost immediate and high citation. Using the journal's average citation impact instead of the actual article impact is tantamount to grading by the prestige of the journal involved. While expedient, it is dangerous. Although journal assessments are important, evaluation of faculty is a much more important exercise that affects individual careers. Impact numbers should not be used as surrogates except in unusual circumstances.¹²

1. Gross PLK, Gross EM. College libraries and chemical education. Science 1927;66:385-9. [Free Full Text]
2. Garfield E. Citation indexes for science: a new dimension in documentation through association of ideas. Science 1955;122:108-111. [Free Full Text]
3. Garfield E. Which medical journals have the greatest impact? Ann Intern Med 1986;105:313-20.
4. Garfield E. Why are the impacts of the leading medical journals so similar and yet so different? Item-by-item audits reveal a diversity of editorial material. In: Essays of an information scientist. Vol 10. Philadelphia: ISI Press, 1987:7-13.
5. Fleischmann M, Pons S, Hawkins M. Electrochemically induced nuclear fusion of deuterium. Journal of Electroanalysis Chemistry Interface 1989:261:301-8.
6. Aboulker JP, Smart AM. Preliminary analysis of the Concorde trial. Lancet 1993;341:889-90. [Medline]
7. Marshall BJ, Warren JR. Unidentified curved bacilli in the stomach of patients with gastritis and peptic ulceration. Lancet 1984;i:1311-5.
8. Lowry OH, Rosebrough NJ, Farr AL, Randall RJ. Protein measurement with the folin phenol reagent. Journal of Biological Chemistry 1951;193:265-75. [Free Full Text]
9. Van Trigt AM, de Jong-van den Berg LT, Voogt LM, Willems J, Tromp TF, Haaijer-Ruskamp FM. Setting the agenda: does the medical literature set the agenda for articles about medicines in the newspapers? Soc Sci Med 1995;41:893-9.
10. Garfield E. How ISI selects journals for coverage: quantitative and qualitative considerations. In: Essays of an information scientist. Vol 13. Philadelphia: ISI Press, 1990:185-93.
11. Lock SP. "Journalology": are the quotes needed? In: Garfield E. Essays of an information scientist. Vol 13. Philadelphia: ISI Press, 1990:19-24.
12. Garfield E. How to use citation analysis for faculty evaluations, and when is it relevant? Parts 1 and 2. In: Essays of an information scientist. Vol 6. Philadelphia: ISI Press, 1984:354-72.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Relevant Articles

Origin and funding of the most frequently cited papers in medicine: database analysis

Nikolaos A Patsopoulos, John P A Ioannidis, and Apostolos A Analatos
BMJ 2006 332: 1061-1064. [Abstract] [Full Text] [PDF]

Impact factors need to be improved

Robert West
BMJ 1996 313: 1400. [Extract] [Full Text]

This article has been cited by other articles:

• Conen, D., Torres, J., Ridker, P. M (2008). Differential Citation Rates of Major Cardiovascular Clinical Trials According to Source of Funding: A Survey From 2000 to 2005. Circulation 118: 1321-1327 [Abstract] [Full text]  
• Holden, G., Barker, K., Covert-Vail, L., Rosenberg, G., Cohen, S. A. (2008). Does Social Work Abstracts Work?. Research on Social Work Practice 18: 487-499 [Abstract]  
• Falagas, M. E., Kouranos, V. D., Arencibia-Jorge, R., Karageorgopoulos, D. E. (2008). Comparison of SCImago journal rank indicator with journal impact factor. FASEB J. 22: 2623-2628 [Abstract] [Full text]  
• Stange, K. C. (2007). In This Issue: Clinical Diagnosis and Management. Ann Fam Med 5: 290-291 [Full text]  
• Chew, M., Villanueva, E. V, Van Der Weyden, M. B (2007). Life and times of the impact factor: retrospective analysis of trends for seven medical journals (1994-2005) and their Editors' views. JRSM 100: 142-150 [Abstract] [Full text]  
• Holden, G., Rosenberg, G., Barker, K., Onghena, P. (2006). An Assessment of the Predictive Validity of Impact Factor Scores: Implications for Academic Employment Decisions in Social Work. Research on Social Work Practice 16: 613-624 [Abstract]  
• Patsopoulos, N. A, Ioannidis, J. P A, Analatos, A. A (2006). Origin and funding of the most frequently cited papers in medicine: database analysis. BMJ 332: 1061-1064 [Abstract] [Full text]  
• Hakkalamani, S., Rawal, A., Hennessy, M. S., Parkinson, R. W. (2006). The impact factor of seven orthopaedic journals: FACTORS INFLUENCING IT. J Bone Joint Surg Br 88-B: 159-162 [Full text]  
• Patsopoulos, N. A., Analatos, A. A., Ioannidis, J. P. A. (2005). Relative Citation Impact of Various Study Designs in the Health Sciences. JAMA 293: 2362-2366 [Abstract] [Full text]  
• Gami, A. S., Montori, V. M., Wilczynski, N. L., Haynes, R. B. (2004). Author self-citation in the diabetes literature. CMAJ 170: 1925-1927 [Abstract] [Full text]  
• IOANNIDIS, J. P. A. (2004). Global estimates of high-level brain drain and deficit. FASEB J. 18: 936-939 [Abstract] [Full text]  
• Pechlaner, H., Zehrer, A., Matzler, K., Abfalter, D. (2004). A Ranking of International Tourism and Hospitality Journals. Journal of Travel Research 42: 328-332 [Abstract]  
• Hunter, J. M. (2004). Editorial III: The latest changes ... no more shorts. Br J Anaesth 92: 7-7 [Full text]  
• Berghmans, T., Meert, A. P., Mascaux, C., Paesmans, M., Lafitte, J. J., Sculier, J. P. (2003). Citation indexes do not reflect methodological quality in lung cancer randomised trials. Ann Oncol 14: 715-721 [Abstract] [Full text]  
• Callaham, M., Wears, R. L., Weber, E. (2002). Journal Prestige, Publication Bias, and Other Characteristics Associated With Citation of Published Studies in Peer-Reviewed Journals. JAMA 287: 2847-2850 [Abstract] [Full text]  
• Mullner, M., Matthews, H., Altman, D. G. (2002). Reporting on Statistical Methods To Adjust for Confounding: A Cross-Sectional Survey. ANN INTERN MED 136: 122-126 [Abstract] [Full text]  
• Gastel, B. (2001). Assessing the impact of investigators' work: beyond impact factors/Evaluer l'influence des travaux de recherche au dela des facteurs d'impact. Canadian J. Anesthesia 48: 941-945 [Full text]  
• Fassoulaki, A., Sarantopoulos, C., Papilas, K., Patris, K., Melemeni, A. (2001). Academic anesthesiologists' views on the importance of the impact factor of scientific journals: a North American and European survey : L'opinion des universitaires en anesthesiologie sur l'importance du facteur d'impact des revues scientifiques : une enquete nord-americaine et europeenne. Canadian J. Anesthesia 48: 953-957 [Abstract] [Full text]  
• Garcia-Rio, F., Serrano, S., Dorgham, A., Alvarez-Sala, R., Ruiz Pena, A., Pino, J.M., Alvarez-Sala, J.L., Villamor, J. (2001). A bibliometric evaluation of European Union research of the respiratory system from 1987-1998. Eur Respir J 17: 1175-1180 [Abstract] [Full text]  
• Gagnon, R. E., Macnab, A. J., Blackstock, D. (2001). An inventory of Canadian anesthesiology. Human research from 1995 through 1999. Canadian J. Anesthesia 48: 452-458 [Abstract] [Full text]  
• Hardy, J.-F. (2001). The Canadian Journal of Anesthesia: the millennium edition/Le Journal canadien d'anesthesie : l'edition du millenaire. Canadian J. Anesthesia 48: 1-5 [Full text]  
• Whitehouse, G H (2001). Citation rates and impact factors: should they matter?. Br. J. Radiol. 74: 1-3 [Full text]  
• Van Fleet, D. D., McWilliams, A., Siegel, D. S. (2000). A Theoretical and Empirical Analysis of Journal Rankings: The Case of Formal Lists. Journal of Management 26: 839-861 [Abstract]  
• Brambrink, A. M., Ehrler, D., Dick, W. F. (2000). Publications on paediatric anaesthesia: a quantitative analysis of publication activity and international recognition. Br J Anaesth 85: 556-562 [Abstract] [Full text]  
• Garfield;, E., Parkin, P. C., Birken, C. (2000). In Which Journals Will Pediatricians Find the Best Evidence for Clinical Practice?. Pediatrics 106: 377-377 [Full text]  
• Stern, R. S., Arndt, K. A. (2000). Top-Cited Dermatology Authors Publishing in 5 "High-Impact" General Medical Journals. Arch Dermatol 136: 357-361 [Abstract] [Full text]  
• Gagnon, R. E., Macnab, A. J., Gagnon, F. A. (2000). A quantitative ranking of Canada's research output of original human studies for the decade 1989 to 1998. CMAJ 162: 37-40 [Abstract] [Full text]  
• Joseph, K. S., Hoey, J. (1999). CMAJ's impact factor: room for recalculation. CMAJ 161: 977-978 [Full text]  
• Hernandez-Borges, A. A., Macias-Cervi, P., Gaspar-Guardado, M. A., de Arcaya, M. L. T.-A., Ruiz-Rabaza, A., Ormazabal-Ramos, C. (1999). Assessing the Relative Quality of Anesthesiology and Critical Care Medicine Internet Mailing Lists. Anesth. Analg. 89: 520-520 [Abstract] [Full text]  
• Higginson, I. J, McGregor, A. M (1999). The impact of palliative medicine?. Palliat Med 13: 273-274  
• Birken, C. S., Parkin, P. C. (1999). In Which Journals Will Pediatricians Find the Best Evidence for Clinical Practice?. Pediatrics 103: 941-947 [Abstract] [Full text]  
• Anonymous, (1997). UK paediatric clinical research under threat. Arch. Dis. Child. 76: 1-3 [Full text]