Origin and funding of the most frequently cited papers in medicine: database analysis
BMJ 2006; 332 doi: https://doi.org/10.1136/bmj.38768.420139.80 (Published 04 May 2006) Cite this as: BMJ 2006;332:1061
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
EDITOR-Why are some papers cited again and again, while for others, you really wonder why nobody ever recognized it?
Scientific papers with a high impact have been investigated in recent years [1]. The paper investigated in this letter [2] was destined to take a course of its very own. It is probably the most cited paper in the history of health science, and it is by far the most cited paper in neuroscience, old age psychiatry and geriatrics.
The Mini Mental State Examination (MMSE) [2] is a brief, quantitative measure of cognitive status in adults. It has been used as a screening instrument to estimate the severity of cognitive impairment - at a given point of time, to follow the course of cognitive changes in an individual over time, and to document an individual’s response to treatment.
The standard MMSE published in 1975 is still in use, with only minor subsequent modifications done by the authors. The MMSE has demonstrated validity and reliability in psychiatry, neurology, geriatrics, as well as in other medical specialities.
The MMSE has been translated into more than 35 authorized foreign language editions.
A bibliometric measurement was done in commercial databases provided by my university library. All citations of the paper between 1975 and 2006 were downloaded from Web of Knowledge (Institute for Scientific Information (ISI)).
Analysis of the frequency of citations per year and distribution of specialities were done by the programmes provided in the Web of Knowledge.
As of November 2006 there were found 19,243 citations on the article since 1977. Frequency of citations per year is shown in figure 1. This shows an exponential increase in the lifetime of the rating scale, which is more than thirty years. The distribution of specialities in frequency of citations is not surprisingly that clinical neurology (5900) and psychiatry (5700) and geriatrics (4500) are the most prominent with equally comparable numbers of citations. Clinical psychology and internal medicine are the succeeding specialities with approximately 1500 citations each. Many papers apply to several specialities.
The Mini Mental State Examination (MMSE) is one of the most ever cited research papers in medical science.
Conclusion:
The paper by Marshal Folstein and colleagues is a very highly cited paper and probably the most frequently cited ever in medicine. It will most likely be that for many years to come. Just to add prosperity to this prophecy, I quote it here once more [2]. I tried to explore if there were reasons for the fate of this paper. Certain elements are obvious, but that it was merely pure chance was not found to be the case. More likely the explanation is that you have to be cited in the right place, in the right time, and probably also by the right people. The “across speciality” use of the scale explains a great part of its success.
Recommendations for further research in this field requires that there will be highly cited papers in the future. Young scientist should therefore consider to create (or get!) an idea for a very simple questionnaire or manoeuvre. Make it brief. Make it useful. Give it an appealing name, with implication of shortness, quickness, or simply that it will make your life as a doctor easier. And finally, these recommendations were never in any way proven to be successful. I am the living proof of that.
PS Professor Marshal F. Folstein, Tufts University School of Medicine, Boston, MA, USA is still working and publishing [3]
Reference List
1. Patsopoulos NA, Ioannidis JP, Analatos AA. Origin and funding of the most frequently cited papers in medicine: database analysis. BMJ 2006;332:1061-4.
2. Folstein MF, Folstein SE, McHugh PR. "Mini-mental state". A practical method for grading the cognitive state of patients for the clinician. J.Psychiatr.Res. 1975;12:189-98.
3. Qiu WQ, Folstein MF. Insulin, insulin-degrading enzyme and amyloid-beta peptide in Alzheimer's disease: review and hypothesis. Neurobiol.Aging 2006;27:190-8.
Competing interests:
None declared
Competing interests: No competing interests
We are afraid that Dr. Michelson’s comments reflect mostly incomplete
reading of our paper and the accompanying appendix. The claim that
perhaps “exciting new research avenues that are being opened by academic
researchers is attracting the [industry] funding” is unlikely to be true.
Simply looking at the type of interventions of the 77 highly-cited
randomized trials, almost all of them examined interventions largely
developed by the industry. Only two trials examined purely changes in
lifestyle and ventilation with lower tidal volumes and another 3 looked
into tighter control of disease with already established drug
interventions or use of an effective intervention in an earlier stage of
the disease. The remaining 94% were typical drug, supplement, or medical
device trials – technologies that are rarely discovered and/or developed
in university labs nowadays. We have also previously shown that the
strongest predictor of a promising basic science finding moving into
randomized trials is the involvement of industry authors in the basic
research [1].
Our study did not aim to investigate specifically the association of
industry funding and effectiveness of the experimental interventions. Our
design is not optimally powered for that question. These are the most-
cited trials, so one expects a preponderance of “positive” results and few
“negative” results to contrast against. Indeed, amongst the 77 trials,
there were 82 independent comparisons of interventions: 69 proved
effective, while 13 proved ineffective or harmful. The percentage of
industry funding was 87% and 69%, respectively, leading to an odds ratio
of 2.96 (95% CI: 0.75-11.7), in full agreement with previous empirical
investigations [2].
Finally, Dr. Michelson certainly misinterpreted our discussion about
medical journals. Conversely, we stated that these journals want to
promote excellence and to attract articles of better quality. This is
common sense. The problem is how best to achieve and safeguard this
attitude in the current research and funding environment.
1. Contopoulos-Ioannidis DG, Ntzani E, Ioannidis JP. Translation of
highly promising basic science research into clinical applications. Am J
Med 2003;114:477-84.
2. Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical
industry sponsorship and research outcome and quality: systematic review.
BMJ 2003;326:1167-70.
Competing interests:
None declared
Competing interests: No competing interests
While I applaud the authors of this study on exploring an important
facet of how medical research is getting funded, the conclusions they draw
are overly broad, speculative, and not actually supported by their data.
They seem to have confused association with causation as they state that
the increased funding of research by industry is determining the research
agenda. Although that is possible, it is also possible that exciting new
research avenues that are being opened by academic researchers is
attracting the funding, rather than the reverse. A good example of this is
the now-frequent transition of academic researchers from academia to
entrepreneurs with their new discoveries. This is not a value judgment as
to whether this is a good thing, but it does not support the contention of
the authors that “Clinical research is dictated by the need to promote
products of industry”.
There is also the implication by the authors that the research
results reported in industry-funded studies is more likely to show
positive effects, and they quote an earlier study supporting this.
However, the authors actually collected the data necessary to examine this
question yet did not mention that aspect of their findings in their own
Results. I can only assume that they did not find such an association
between funding sources and study results. If this is true, then the
authors are remiss in not explicitly reporting on this.
Finally, the authors do a great disservice to the leading medical
journals by speculating that they publish articles that are more likely to
be “popular” than meticulous and unbiased. If anything, the leading
journals have gone out of their way to proclaim that they will not follow
such a path, precisely for the reasons cited in the current study.
I am disappointed in the alarmist tone of this article. Despite
addressing a serious concern in academic medicine, its unsupported
speculations and allegations detract from what should be a rigorous
discussion.
Competing interests:
None declared
Competing interests: No competing interests
On citing scientific papers properly
Madam,
In his delightful comments(1) on an earlier paper in the BMJ on
citation analysis (2) Dr Nilsson draws attention to the (probably) most
frequently cited paper in recent medical history(3). His suggestion for
young scientist who think of writing a successful paper (Give it an
appealing name, with implication of shortness, quickness, or simply that
it will make your life as a doctor easier) could perhaps be paraphrased
succinctly: Write a method paper! Method papers are routinely cited more
frequently than theoretical or empirical studies, in medical science and
other subjects, a phenomenon previously described (4).
Dr Nilsson relied on the excellent Web of Knowledge database (WoK)
(Institute for Scientific Information (ISI)) to calculate that the paper
by Folstein et al. (3) received some amazing 19,250 citations since 1977.
I would like to point out that this figure should be increased by at
least 500, this being the number of papers citing Folstein incorrectly.
Among incorrect references are 300+ from papers which get the page or
volume wrong (usually but not exclusively ‘198’ or ‘21’ respectively),
many which give an incorrect source journal (incl. suggestions that the
paper was published in JAMA) or identify Folstein’s paper as having been
published as late as 1997.
The WoK has a powerful ‘analyze’ facility which allows to identify
institutions which let, in this case, incorrect references slip through
their editing process. Each of these institutions has a least ten papers
with an incorrect reference to Folstein’s paper:
Univ Calif Los Angeles; Harvard Univ; Johns Hopkins Univ; Univ Cambridge;
Univ Pittsburgh; Australian Natl Univ; Univ Texas; Univ Washington; Univ
Toronto; Columbia Univ.
Among periodicals which have a least 10 papers with an incorrect
citation are:
Neurology; Journal of the American Geriatrics Society; International
Journal of Geriatric Psychiatry; Dementia and Geriatric Cognitive
Disorders; Journal of Clinical and Experimental Neuropsychology; Archives
of Neurology; Aging & Mental Health; Aging Neuropsychology and
Cognition; European Journal of Neurology.
A further analysis identifying authors who cite the Folstein
incorrectly and later authors who then just copied the faulty reference
without going to the original publication (or at least checking the
reference on PubMed) would be possible.
Sadly, Nilsson’s reference will not add to the citation count for
Folstein’s paper, as references in BMJ’s Rapid Responses are not visible
on the Web of Knowledge. A variant of the FUTON Bias(5), perhaps?
Respectfully yours,
Reinhard Wentz
(1) http://www.bmj.com/cgi/eletters/332/7549/1061#149065
(2) Patsopoulos NA, Ioannidis JP, Analatos AA. Origin and funding of the
most frequently cited papers in medicine: database analysis. BMJ
2006;332:1061-4.
(3) Folstein MF, Folstein SE, McHugh PR. "Mini-mental state". A practical
method for grading the cognitive state of patients for the clinician.
J.Psychiatr.Res. 1975;12:189-98.
(4) Peritz BC.Are methodological papers more cited than theoretical or
empirical ones? The case of sociology. Scientometrics 1983; 5 :211-18
(5) http://en.wikipedia.org/wiki/FUTON_bias
Competing interests:
None declared
Competing interests: No competing interests