Inadequate post-publication review of medical research
BMJ 2010; 341 doi: https://doi.org/10.1136/bmj.c3803 (Published 11 August 2010) Cite this as: BMJ 2010;341:c3803All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
At the outset I would like to congratulate BMJ & Prof. David L
Schriger, Prof. Douglas G Altman for a brilliant Editorial on Inadequate
post-publication review of medical research.
Peer review process in publication is very important quality factor
for the journals as well as the authors who publish in peer reviewed
journals. Though the peer review process in scholarly activities has
improved over period of time, it needs lot of improvement both at pre as
well as post publication phase. Many new innovations in scholarly
publication are being tried world over. Nature2 had tried open review
experiment in 2006 and now Shakespeare Quarterly3 tried an open review
method of peer review by MediaCommons Press in pre publication phase.
Nature invited authors between 1 June and 30 September 2006, after initial
editorial assessment for open review at public domain along with standard
peer review. This lead to increase interest among scientist but only few
scientists commented on the open articles. During the whole exercise, 74%
agreed that their paper had been improved.
As far as post publication review is concerned, very few journal have
excellent system like BMJ- Rapid responses, Annals of Internal Medicine-
Rapid response. Many authors do not respond to comments of rapid
responses4 which is not good practice for healthy scientific community
while authors usually respond to all comments of reviewers before
publication. There is need to have some kind of commitment or agreement on
the part of authors to respond to these comments within a stipulated time
frame. This may be prescribed by the journals or through International
Committee of Medical Journal Editors or through Committee on Publication
Ethics(COPE).
1.Prof. David L Schriger, Prof. Douglas G Altman. Inadequate post-
publication review of medical research BMJ 2010;341:c3803
2.Despite enthusiasm for the concept, open peer review was not widely
popular, either among authors or by scientists invited to comment.
Overview: Nature's peer review trial Nature December 2006 (accessed online
http://www.nature.com/nature/peerreview/debate/nature05535.html on
25.08.10)
3.MediaCommons Press (accessed online
http://mediacommons.futureofthebook.org/mcpress/ShakespeareQuarterly_New...
on 25.08.10)
4.Peter C Gøtzsche et.al. Adequacy of authors replies to criticism
raised in electronic letters to the editor: cohort study BMJ
2010;341:C3926
Competing interests:
Editorial Board Member of 4 journals
Competing interests: No competing interests
I am grateful for Schriger and Altman raising the issue of post
publication review1. Some years ago a high impact specialty journal
published the results of an RCT that I felt was flawed in its methodology
and data interpretation. I submitted a letter with my comments through the
approved channel. Neither the receipt of the original letter nor a follow
up enquiry were acknowledged. My letter was never published. A few months
later a further paper based on the same RCT was published in a lower
impact, more specialist journal. My comment letter was published this time
but the authors did not respond. When the results of the study were used
to market the technology which appeared more effective, I again questioned
the results with a local sales representative of the manufacturer, who
said he would make enquiries. I heard nothing. The manufacturer in
question is a large international company which not only sponsored the
trial but also is a major financial backer of the annual conference of the
professional society that publishes the first journal I wrote to.
As a clinical researcher, systematic reviewer and guideline developer I
recognise the value of post publication criticism and debate. Could a
condition of publication be an obligation to respond to criticism?
1.Schriger DL and Altman DG. Inadequate post-publication review of
medical research. BMJ 2010;341:c3803
Competing interests:
None declared
Competing interests: No competing interests
Perfect peer reviews are rare. I perform peer reviews almost every
week. I try to do my best but I can't be an expert at all aspects of the
studies. Even if we were 3 or 4 reviewers for each paper, and even if we
were paid, we would fail to discover some flaws. I have seen too many
uncritical citations of flawed papers.
Editors should promote post-publication reviews and the reviews
should be opened when the on line versions of papers are opened, like BMJ
Rapid Response.
Competing interests:
I am associate editor for 4 journals and a peer reviewer for BMJ
Competing interests: No competing interests
I fully appreciate the drawbacks of pre-publication peer review. The
solution to the problem lies in strengthening this aspect of publication
and not in ridiculing it. Peer-review has several advantages other than
the obvious ones that are difficult to quantify. For example, the very
existence of pre-publication peer review might deter many poor quality
papers at the point of origin.
End users know how to categorize the articles. If I am hoping to make
significant changes in a common area of my clinical practice, I would no
doubt read the papers in top journals only and not those published in
obscure journals. On the other hand, I might happily consider the findings
of just about any paper in any journal when dealing with a rare syndrome.
Similarly, regional journals are no less important when it comes to issues
that are locally relevant. An article in a low impact factor journal might
generate somebody’s interest and lead to a more substantive work in the
field. Thus, every category of journal offers something that is relevant
to somebody. Fewer papers (presumably only in top journals) would deprive
us of all these benefits while unrestricted free-for-all information and
rounds of letters would be complete chaos. If original articles are not
perfect, why should letters and post-publication comments be assumed to be
perfect?
Strengthen peer review prior to publication (e.g. paid reviewers,
timely review) and then leave things to the wise end-users.
Competing interests:
I do not have high-quality original papers as a lead author (i.e. corresponding author)
Competing interests: No competing interests
Schriger & Altman make an important point in highlighting the
inherent tension editors may feel in publishing critical responses to
published items. The COPE (Committee on Publication Ethics) Code of
Conduct for editors states that 'Cogent criticisms of published work
should be published unless editors have convincing reasons why they cannot
be'. Cases and queries received by COPE by disgruntled commentators and
authors in the last couple of years suggest that some editors are falling
short of this standard. Yet we also hear from editors facing unreasonable
demands to publish inappropriate responses such as personal attacks on the
authors. Maybe some guidelines (for editors, commentators and authors)
would be helpful? If so, this is something that COPE might consider.
Competing interests:
Chair of COPE
Competing interests: No competing interests
Authors have too narrow a view of post-publication review
David Schriger and Doug Altman bemoan the inadequacy of post-publication review, but surely they have too limited a concept of post-
publication review. They don't explicitly define post-publication
review, but they imply that it is the process of people writing to
criticise published reports and the authors responding. This is much too
narrow a view and is making the common mistake of confusing the publishing
of science with its doing.
I would define post-publication review as the process whereby
scientists and others decide whether a piece of work matters or not. I
suggest that this doesn't happen much through debate in the
correspondence pages of journals, but rather through scientists and other
consumers of research recommending others to pay attention to a piece of
research, conducting other studies off the back of it, absorbing it into
systematic reviews, beginning to act on its conclusions, throwing it in
the bin, and taking a thousand other actions that constitute the "market of ideas."
Why waste your time writing to journals, particularly when, as
Schriger and Altman imply, "many research studies don't
matter much and most don't matter at all"?
Competing interests:
I'm a former editor of the BMJ, a friend of Doug Altman, on the board of the Public Library of Science, and increasingly convinced that the processes of publishing science are broken.
Competing interests: No competing interests