Inadequate post-publication review of medical research

BMJ 2010; 341 doi: http://dx.doi.org/10.1136/bmj.c3803 (Published 11 August 2010) Cite this as: BMJ 2010;341:c3803
  1. David L Schriger, professor of emergency medicine1,
  2. Douglas G Altman, director and professor of statistics in medicine2
  1. 1Emergency Medicine Center, School of Medicine, University of California Los Angeles, 924 Westwood Blvd Suite 300, Los Angeles, CA 90024-2924, USA
  2. 2Centre for Statistics in Medicine, University of Oxford, Oxford OX2 6UD
  1. schriger{at}ucla.edu

    A sign of an unhealthy research environment in clinical medicine

    Imagine a vibrant field of scientific inquiry. Researchers focus on solving the most urgent uncertainties of the discipline and publish papers that, guided by reporting guidelines1 and further improved by pre-publication peer review, provide comprehensive accounts of their methods, findings, and limitations. The research community, keen to advance the field, engages in an active dialogue regarding the validity and implications of each new paper. Post-publication critique, as the final arbiter of the meaning of each new communication, is no less important than the earlier phases and is a sign of a healthy scientific community, a community actively working to move the field forward.2 3 4

    In the linked study (doi:10.1136/bmj.c3926),5 Gøtzsche and colleagues’ finding that authors of BMJ articles are reluctant to respond to criticisms submitted as rapid responses reinforces the finding of the few previous studies,4 6 7 which found consistent evidence that all aspects of post-publication review are wanting in medical research. Most research articles in medical journals receive no critique, and, for the minority that do, authors often do not reply or reply but do not respond to the criticism. Instead of illuminating the factors that would help the community determine the validity and importance of a paper, the current post-publication review process typically leaves key concerns unresolved.

    Given that science is fallible, what should we conclude about a field of scientific inquiry where post-publication review is lacking? Is the problem with the post-publication peer review process, the prevailing culture, or the publications themselves? The table lists factors that could contribute to a suboptimal post-publication review process. Because peer review is imperfect,8 and evidence shows that most published papers have important flaws in methods9 or reporting,10 the first explanation—that papers are of such high quality that no commentary is needed—can be quickly dismissed.

    Reasons for the absence of adequate post-publication peer review

    View this table:

    We can only speculate about the extent to which the low volume of letters is a function of the papers (which are often written to promote the authors or a product rather than move science forward and benefit patients), the readers (errors are not detected or readers who see them don’t bother writing), or the editors (who resist publishing critical letters that impugn their article selection and review process). Today’s research contributes to tomorrow’s systematic reviews, so an important flaw in a publication should be highlighted whenever identified.11 It is therefore regrettable that journals discourage letters through needless constraints on length, the allowed time window, the number of letters they will publish, and their refusal to publish several rounds of letters.11 12

    Journals have been exercised about competing interests of authors and reviewers. Yet journals have a strong conflict of interest regarding letters to the editor because publishing criticisms of journal articles suggests that the editors are not doing their job and may lower the prestige of the journal, and handling correspondence requires journal resources. There is a case for an independent letters editor.

    If logistics were the sole problem we would expect that web based rapid response features that remove many of the logistical difficulties of publishing a critique would be popular. They are not widely used by (largely) paper based journals other than the BMJ,7 and when they are used, they often fail to get complete responses from the paper’s authors,5 which suggests that logistics are not the sole cause of inadequate post-publication review.

    The inadequacy of authors’ responses to criticism suggests that authors feel no obligation to respond to reasoned criticism, and letter writers may fear that they will be perceived as picky or anti-collegial for pointing out flaws. Such a culture impedes the progress of science by stifling the open communication that makes the literature self correcting and is essential to the scientific process.2

    Finally, the volume and quality of scientific papers may contribute to the problem—a mountain of poor quality unfocused literature has left its readership fatigued, numb, and passive. Each year more papers are published than the year before (about 500 000 research papers were added to Medline in 2009), but the number of letters stays the same.12 Each new paper is another monologue added to the heap. Few read it and fewer care. Errors remain unnoticed or un-noted, and no one seems terribly bothered.

    The solution to the absence of effective post-publication reviews does not lie within its mechanisms; it requires a fundamental reworking of what research is performed, how it is presented, and how it is assimilated into current knowledge. We need fewer papers that are of higher quality and importance. We also need a change in culture to value public discussion if we are to re-engage the medical research community in the kind of post-publication review process that patients deserve.4 9


    Cite this as: BMJ 2010;341:c3803


    • Research, doi:10.1136/bmj.c3926
    • Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

    • Provenance and peer review: Commissioned; not externally peer reviewed.