Intended for healthcare professionals

Research

Origin and funding of the most frequently cited papers in medicine: database analysis

BMJ 2006; 332 doi: https://doi.org/10.1136/bmj.38768.420139.80 (Published 04 May 2006) Cite this as: BMJ 2006;332:1061
  1. Nikolaos A Patsopoulos, research associate1,
  2. John P A Ioannidis, professor (jioannid{at}cc.uoi.gr)1,
  3. Apostolos A Analatos, medical student2
  1. 1 Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece
  2. 2 University of Thessaly School of Medicine, Larissa, Greece
  1. Correspondence to: J P A Ioannidis
  • Accepted 2 February 2006

Abstract

Objective To evaluate changes in the role of academics and the sources of funding for the medical research cited most frequently over the past decade.

Design Database analysis.

Data sources Web of Knowledge database.

Methods For each year from 1994 to 2003, articles in the domain of clinical medicine that had been cited most often by the end of 2004 were identified. Changes in authors' affiliations and funding sources were evaluated.

Results Of the 289 frequently cited articles, most had at least one author with a university (76%) or hospital (57%) affiliation, and the proportion of articles with each type of affiliation was constant over time. Government or public funding was most common (60% of articles), followed by industry (36%). The proportion of most frequently cited articles funded by industry increased over time (odds ratio 1.17 per year, P = 0.001) and was equal to the proportion funded by government or public sources by 2001. 65 of the 77 most cited randomised controlled trials received funding from industry, and the proportion increased significantly over time (odds ratio 1.59 per year, P = 0.003). 18 of the 32 most cited trials published after 1999 were funded by industry alone.

Conclusion Academic affiliations remain prominent among the authors of the most frequently cited medical research. Such research is increasingly funded by industry, often exclusively so. Academics may be losing control of the clinical research agenda.

Introduction

The debate over academic medicine argues that it is in crisis and that urgent measures are needed to promote and revitalise it.1 2 One area of concern is whether academic medicine is losing control of the medical research agenda. Academic doctors' influence on research in basic biology has decreased,3 4 but do academic doctors still control medical research? Medical research may depend on funding from the private sector, in particular from biotechnology and drug companies.5 6 This funding may lead to conflicts of interest about the results of medical research.5 6 7 8 9

Despite the importance of these issues, no quantitative evidence is available. We need empirical data on how many leading studies in medicine have authors who are academics. We also need empirical data on the extent of funding by industry. Have changes occurred in the participation of academics and the funding of studies during the past decade?

Our analysis was one of the evidence tasks (systematic reviews and surveys addressing problems of academic medicine) undertaken as part of the International Campaign to Revitalise Academic Medicine (ICRAM).3 We analysed the affiliations of authors and the funding sources of articles in the domain of clinical medicine that had received the highest number of citations according to the essential science indicators module of the Web of Knowledge database. Our main aim was to see whether the impact of academic institutions and industry has changed during the past decade. Citations do not reflect fully the quality of a paper, but they are a measure of the impact of research.10 Papers that are cited frequently have a major impact on scientific debate and the evolution of scientific thinking, even if citations are not always positive and may even be critical.

Methods

Identification of the most frequently cited papers

We downloaded the most frequently cited papers in clinical medicine of each year since 1994 from the essential science indicators module of the Web of Knowledge produced by the Institute for Scientific Information (ISI). Articles were ranked according to how many times they were cited by any journal indexed by ISI between 1 January 1994 and 31 December 2004. Clinical medicine is one of 21 scientific domains catalogued by ISI and covers all medical sciences and subspecialties. ISI categorises articles across the 21 domains according to the journal of publication; for articles published in multidisciplinary journals, each article is categorised in the domain that most of its citations are derived from.

We screened the 1846 articles published since 1994 that had been cited more than 325 times by the end of 2004. We retained the 30 most cited articles for each year to analyse changes in affiliations and funding over time. We analysed papers from 2002-4 together (23 articles from 2002 and seven from 2003; no article published in 2004 had been cited more than 325 times by the end of the year).

Even though most studies with group authorship are classified appropriately, essential science indicators may have failed to classify a few of the earlier studies correctly. Articles with group authorship are usually intervention studies (mostly randomised controlled trials) and meta-analyses.11 Therefore, we also performed a more detailed screen in the Web of Science of the New England Journal of Medicine, JAMA, Lancet, and BMJ for other articles that had been cited 325 times in the same period of time. The first three journals publish about 95% of the most frequently cited randomised trials,12 and together with the BMJ they should also cover most of the frequently cited meta-analyses. This additional search yielded another 19 articles in the Web of Science module that had not been identified originally by essential science indicators. Thus we analysed 289 articles in total (see appendix on bmj.com).

Data extraction

From the full text versions of the 289 articles we extracted data on the journal, year of publication, volume and pages, type of study, country (or countries) of origin, authors' affiliations (university, hospital, professional society, government, nongovernment research institute, drug or biotechnology company (for profit), other or unknown; more than one category was allowed), and funding (government or public sources, university, private organisation or foundation, professional society, drug and biotechnology company (for profit), and other or unknown; more than one category was allowed). For randomised controlled trials we also recorded whether the tested interventions were predominantly effective, not effective, or harmful.

Fig 1
Fig 1

Proportion of most frequently cited articles published each year according to authors' affiliations. Studies with more than one category of affiliations are counted in all relevant categories

Two independent investigators extracted the data, and discrepancies were discussed to reach consensus. We used the internet to identify addresses and descriptions of institutions and funding sources that were not readily apparent. Remaining discrepancies were settled by discussion with a third investigator.

Analysis

Our main analyses examined whether the most cited articles had changed over time in terms of the academic profile of authors' affiliations and funding from industry.

Descriptive statistics used medians and proportions. We used logistic regression, with year of publication as the independent variable, to model changes over time in the proportion of specific affiliations of authors and the proportion of specific sources of funding. We analysed randomised controlled trials separately because they are the most common type of research with original data that might directly affect clinical practice.

We performed sensitivity analyses: for each year between 1994 and 2001 we retained the top 30 papers on the basis of their citation counts for the first four years (publication year and three subsequent years) rather than total citations until the end of 2004, to give similar follow-up for articles published in different years. Only 39 articles were replaced in the analysis (15%); seven more randomised controlled trials were included and four others were removed; sensitivity analysis did not alter the results (not shown).

We used SPSS 12.0 for Windows to analyse the data, and P values were two tailed.

Results

Type and origin of eligible articles

The table shows the different types of studies analysed.

Different types of most frequently cited papers analysed, 1994-2003

View this table:

The median number of citations ranged from 420 for articles published in 2003 to1409 for articles published in 1994. Half of the articles (145; 50%) had affiliations in the United States only, and almost a third (86; 30%) had affiliations from the US and at least one other country, whereas a minority of articles had affiliations from a single country other than the US (40; 14%) or several countries other than the US (18; 6%). In all papers authors came from developed countries except for two from China and another 14 in which co-authors came from developing countries. Countries of origin did not change significantly over time (not shown).

Authors' affiliations

The proportion of articles with specific types of authors' affiliations did not change (fig 1). Over the decade, 220 (76%) articles had at least one author with a university affiliation and 164 (57%) had at least one author with a hospital affiliation. Authors had government, non-government research institute, industry, and professional society affiliations in 95 (33%), 70 (24%), 85 (29%), and 17 (6%) articles, respectively.

Sources of funding

Over the decade, proportions of the most cited articles funded by drug and biotechnology companies increased (fig 2). In logistic regression, the odds of funding by industry increased 1.17-fold (95% confidence interval 1.06 to 1.28, P = 0.001) per year. More than 50% of funding came from industry in 2001, surpassing funding from government or public sources, which decreased non-significantly over time (odds ratio 0.94 (0.86 to 1.03) per year, P = 0.19). Over the decade, government or public sources, industry, university, private foundations, and professional societies funded 173 (60%), 104 (36%), 29 (10%), 64 (22%), and 54 (19%) papers, respectively. A clear increase was seen only for funding by industry (fig 2).

Fig 2
Fig 2

Proportion of frequently cited articles published each year according to sources of funding. Studies with funding from diverse categories of support are counted in all relevant categories

Affiliations and funding for randomised controlled trials

Most of the frequently cited randomised controlled trials (60 of 77) investigated drugs or biological agents and another five looked at medical devices or stents. Five trials investigated the efficacy of vitamins, four hormone replacement treatment, two lifestyle changes (one of them compared with drugs), and one a surgical intervention. Most trials found that the interventions were effective (67 of 75). One trial found that one of the two interventions was effective, six found that interventions were not effective, and three found that the intervention (hormone replacement therapy) could be harmful.

Randomised controlled trials typically had authors with university (66 of 77; 86%) affiliations or hospital (67; 87%) affiliations, whereas almost half (36; 47%) had authors with industry affiliations. Fewer had government (21; 27%) affiliations, non-government research institute (10; 13%) affiliations, and professional society (1; 1%) affiliations. The author profile did not change significantly (data not shown).

Funding by industry was common in randomised controlled trials and increased over the 10 years. Sixty five of the 77 most frequently cited trials had funding from industry, whereas government and public sources funded only 32 trials, and other sources of funding were rare. The proportion of trials funded by industry increased significantly over time (1.59 (1.17 to 2.15) per year, P = 0.003). Thirty one of the 32 frequently cited trials published after 1999 were funded by industry, 18 exclusively so.

Discussion

The participation of academics in the most cited medical research has remained strong over the past decade, despite the perceived crisis in academic medicine. However, industry funds an increasing proportion of this influential medical research, especially randomised controlled trials, most of which are now funded exclusively by industry.

Limitations

The high citation counts of these papers do not necessarily mean that they represent the best or most important research performed. Most clinical trials are now centred around industry sponsored interventions. Clinical research is dictated by the need to promote products of industry. In this sense, academics may have indeed lost control of the clinical research agenda. The role of industry should be scrutinised further.13 Many important questions in clinical research have no connections with specific products and thus would not be supported by industry.14 Such questions may never be funded or may not be cited frequently once published.

Implications

Funding by industry highlights the importance of building, maintaining, and strengthening healthy and productive partnerships within research.4 5 Advances in biotechnology provide opportunities for collaboration between academics and the private sector, but safeguards are needed to guard against financial conflicts of interest.6 7 8 9

The recent increase in funding for profit may not be solely due to the need of industry to advertise. Although governments have always funded biomedical research, recent cuts in research budgets provide an opening for the private sector. The 2006 budget for research and development in the US (the major contributor to high impact clinical research) increased by just 0.7%, whereas 6% is needed to maintain the current success rate of funding applications.15 16 Together with the steady increase of applications for funding, this shortfall provides an opportunity for industry to fund more research projects with promising commercial uses. Tighter connections between the academic community and industry can be a benefit and a side effect of co-funding programmes (additional funding by a nongovernment body). Canada has recently promoted co-funding models; this has resulted in increased research productivity, but has raised questions about the influence of partnerships with industry on the academic community.17

Other factors may also influence the relationship between academic medicine and its funders. Medical journals convey scientific knowledge to the biomedical community, but they may also be used by industry to promote its interests.18 Trials sponsored by industry are more likely to show favourable outcomes,19 and may have better chances of being published in prestigious journals. Randomised controlled trials usually receive many citations,20 especially if they show “positive” results. Citations strengthen the prestige of the journal, so that it can attract articles of better quality, but journals may wish to publish only those papers that are likely to be cited often, thereby creating a circle of excellence that could become vicious.

The future of academic medicine

Different scenarios have been suggested for the future of academic medicine.21 Our findings are in line with the scenario of “Academic Inc,” with academic medicine evolving into an efficient enterprise that is directed by profit and has strong ties to other profit making corporate structures. Our findings do not agree with a scenario where academic medicine disappears. Participation in influential medical research has remained strong despite the advent of serious competition for funds and studies from research institutes and other (for profit) organisations. However, most influential research originates from the US and other wealthy countries, a situation that is dissonant with a global view of health needs. Few trials are conducted in developing countries, or receive the attention of other investigators.22 Government and the public sector still make major contributions to academic research, despite the rapid increase in industry sponsorship in areas such as randomised controlled trials. Medical research should reflect public needs more closely and the efforts of all of those involved (mainly government, industry, and academia) should be better coordinated.

What is known on this topic

Academics may be losing control of the research agenda and of clinical research at large

What this study adds

Most of the frequently cited papers on medical research in the past decade continue to have authors with academic affiliations

Since 1994, biotechnology and drug companies have provided funding to an increasing proportion of frequently cited studies

Industry funding has become ubiquitous for frequently cited randomised controlled trials, and most of these influential trials have no other sources of funding

Footnotes

  • Embedded Image An appendix with the 289 most frequently cited articles is on bmj.com

    NAP and AAA thank A Germenis for his encouragement. This work was conducted under the auspices of the International Campaign to Revitalise Academic Medicine (ICRAM) as part of the effort to evaluate the evidence base for academic medicine. Members of the ICRAM Working Party are listed on bmj.com/academicmedicine.

  • Contributors JPAI had the original idea for the study and also wrote the protocol, which was commented on by the other two authors. NAP and AAA performed the data extraction and discrepancies were resolved by JPAI. NAP and JPAI performed the statistical analyses. All authors interpreted the results. JPAI and NAP wrote the final manuscript and AAA commented on it. All authors approved the final version. JPAI is guarantor.

  • Funding None.

  • Competing interests None declared.

  • Ethical approval Not required.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
View Abstract