Burden of proof: combating inaccurate citation in biomedical literature

Types of inadequate citation

Just as there is no universally accepted term—misquotation8 versus quotation errors,2 erroneous citation15 versus inadequate citation,12 and so on—there is no universally accepted classification scheme. Steven Greenberg developed a “vocabulary” for the “citation distortions” he encountered in his study of amyloid β, which offers one useful framework, but it primarily describes mechanisms and consequences of select forms of poor citation without exploring underlying causes, which also matter.9 Others discuss “common citation errors” but do not report the methodology for how they arrived at these categories.19 We propose that rigorous, systematic categorisation, a taxonomy and nomenclature of inadequate citation, or both, are important next steps. Box 1 provides a concise overview of some common documented types of mis-citation.

Strategies to mitigate mis-citation

There are few checks and balances on erroneous citation. The first line of defence (pre-submission quality control) is a standard expectation of academic integrity. In the age of “publish or perish,” however, competing interests, such as enormous pressure on investigators to exhibit constant productivity, can make it tempting to cut corners in pursuit of expediency.62124 Market forces are in tune to this. Consensus, for example, is a new, AI powered search engine designed specifically for academics (https://consensus.app/search/). Given a search query, it will produce a list of 5-10 peer reviewed papers with a short synopsis of each. The more widely used ChatGPT has been found to fabricate both facts and legitimate sounding “scholarly sources” to provide a convincing—rather than factual—answer to a prompt.1617 Although these tools are multi-purpose, powerful, and certainly promising adjuvants for literature review, they also exponentially enable the ease and scale of producing inaccurate citations.

The second line of defence is peer review, which currently serves as the major safety net to catch mis-citations after submission. History has shown, however, that this can also be inefficient, inconsistent, and insufficient.6262728 (And, as others point out, peer review may even encourage mis-citation.)20 Moreover, the ultimate responsibility for auditing cited literature should not rest with peer reviewers, but with those who selected the literature to support their claims.

The third line of defence is post-publication review, which is often either inordinately difficult2930 or even directly opposed.31 Although non-replicable primary results have been caught and overturned after publication, there is little drive to do the same for inaccurate citations. This effectively allows them to live on into eternity. Although we are not the first to highlight these problems,1532 we argue that prevailing strategies are insufficient.

New tools in our toolkit

Just as AI might exacerbate erroneous citation, we suggest it could also be part of the solution. It might, for example, eventually be possible for AI programmes to be integrated into manuscript submission portals to confirm that all cited works actually exist. Even more useful, however, will be AI modalities developed to assess citations for accuracy and to flag potential discrepancies for review. This would not replace human review, but something with reasonable sensitivity and high specificity could provide an initial screen that alerts reviewers to instances of potential mis-citation without creating additional work.

Another idea is for journals to ask authors to attest that an article contains no inaccurate citations (box 2), much in the same way that they are expected to make a statement about conflicts of interest.

These ideals are implicit in any submission to a peer reviewed journal, so making them explicit presents no addition burden. The key question, then, is whether such declarations are useful. To answer this, we can compare older to more recent literature as journals have progressively adopted more rigorous standards. Specifically, conflict of interest (COI) statements are a useful analogy to our proposed works cited statement, because when there are no competing interests to declare, they function as an attestation. These statements became increasingly common as concerns grew about the influence of money and corporate interests in research. Estimates for the proportion of journals requiring a COI statement vary by discipline but consistently show a positive trend. Some commonly cited examples include 16% (220 of 1367 highly ranked scientific and biomedical journals) in 1997,33 33% (28 of 84 journals from 12 scientific disciplines) in 2007,34 89.7% (358 of 399 “high impact biomedical journals”) in 2011,35 and 96% (224 of 227 public and occupational health journals) in 2016.36 It comes as no surprise, then, that none of 47 trials on febrile neutropenia published from 1981 to 2000 contained a COI statement.37 (Additionally, only 29 reported that informed consent was obtained and only 22 reported approval of a research ethics committee.) To see how this compares with more recent literature, we reviewed a random sample of 100 research papers published in 2022 in the New England Journal of Medicine, the Journal of the American Medical Association, and The BMJ, finding that 100% of articles included COI statements (and likewise, statements on institutional review board approval and informed consent, when applicable) (see supplementary file on bmj.com).

A COI statement encourages transparency because inadequate disclosure carries consequences. Similarly, a “works cited statement” (box 2) might encourage diligence as it implies that references will be scrutinised, and mis-citation will be penalised. Assuming that most authors operate on good intent (striving for their work to be accurate) or self-interest (striving to avoid delays in publication), or both, the act of formal attestation might inspire greater diligence in crafting a reference list or double checking it for accuracy before submission. Even with inadequate disclosure, however, statements still offer important functionality. A 2014 phase 3 trial, for example, was publicly redacted when it was discovered that the authors lied in their COI statement.38 Here, the COI statement created a clear mismatch between the authors’ words and deeds. This alerted a vigilant reader, provided the journal with irrefutable justification for corrective action, and, ultimately, halted the dissemination of untrustworthy information. Similarly, a works cited statement found to have inadequate or inaccurate disclosure might raise a red flag and lead reviewers or readers to scrutinise a work more closely, potentially to catch other important flaws. If the author’s formal assurances for something as basic as a reference list cannot be taken at face value, what else might they have been mistaken or misleading about?

COI statements enable new inquiries into research bias. Landmark studies have shown, for example, a strong association between pharmaceutical industry funding and likelihood of reporting significant results.3940 Another study analysed 767 clinical trials and found a strong association between failure to disclose informed consent and poor methodological quality.41 A works cited statement, then, would enable researchers to ask similar questions about inaccurate citations, such as whether studies with an inaccurate works cited statement are associated with poor methodological quality or inflated results. Fact checking citation accuracy can be labour intensive, but there is a critical mass of people who do this sort of work already, reflected in the growing literature on mis-citation.1235689192021222324253032

So, declarations do seem to have beneficial value. They are not a perfect, systemic failsafe, but they do provide authors, editors, peer reviewers, and journal readers with additional opportunities to promote essential quality control. Similar cases can be made for the required declaration of institutional review board approval, author contributions, informed consent, and emerging declarations of the role of AI. As thousands of erroneous citations now raise legitimate questions about inaccuracies in published research, there is a clear case for adopting similar safeguards in this arena as well. To make such a declaration meaningful, however, it must be enforced.42

As a final strategy, we propose that journals add two internal questions for reviewers: were any improper citations noted during the review of this paper? And did you recommend the authors to cite any studies in which you are a coauthor or otherwise have a vested interest? If the answer to either question is yes, reviewers must provide specific details, which are sent to the editor. We propose that a submitted study with erroneous citations should be withdrawn from consideration if the errors are pervasive, mischaracterise the background or methods, inappropriately shape interpretation of the results, or otherwise betray a serious lack of expected due diligence. For manuscripts with less egregious mis-citation, we recommend that reviewers and editors still adopt a low threshold to mandate revisions, as “there is no good reason to allow . . . inexact and non-verifiable referencing to pervade scientific literature.”6 We further propose that an editor should closely scrutinise situations in which a reviewer has recommended citation of their own works.20 If already published when the errors are caught, authors should be asked to amend the work (without additional fees29), which is eminently possible in the digital era. Critically, we extend investigators the benefit of the doubt: many inaccurate citations are simply the result of honest mistakes. Thus, this higher standard is primarily meant to deter carelessness and promote good practice. As is the case broadly in medicine, primary prevention is often the best policy.

Looking ahead

Mis-citation has heretofore been inadequately tackled. By acknowledging the fault lines in current practice, prioritising the development of a rigorous and standardised classification scheme for inadequate citation, and codifying accurate literature review into a routine pre-submission declaration, we can strive to better enshrine integrity into medical scholarship. We encourage journals to consider adopting a pre-submission declaration. It has the potential to deter inappropriate manuscript submissions and facilitate correction after publication, with little added cost or inconvenience. Over time, the higher standard might also instil greater respect among the next generation of young investigators for this fundamental pillar of scientific inquiry. Most importantly, if it yields improvement in overall citation quality, this will help the scientific literature better function as an accurate web of knowledge.