Intended for healthcare professionals
BMJ critical event audit: outcomes
The independent Statins Review that was convened to consider issues around the publication of two articles, by Abramson et al. and Malhotra, recommended that “The BMJ editorial staff should implement a significant event audit in relation to the need for the correction. The aim of the audit would be to try and identify what would need to have been in place to ensure that the correction was made in a more timely fashion.” The editors undertook an end-to-end critical event audit of the handling of the two papers and their associated corrections. This document describes this e BMJ’s audit and its recommendations.
This audit was completed by current and former senior editorial staff of the journal, with oversight of the Editor-in-Chief, Fiona Godlee. The following staff contributed:
- Theodora Bloom, Executive Editor (since June 2014)
- Tony Delamothe, Deputy Editor
- Fiona Godlee, Editor-in-Chief
- Trevor Jackson, Head of News and Comment
- Karl Sharrock, Lead Technical Editor, News and Comment
- Jane Smith (Deputy Editor until end 2012 and Secretary to the Independent Panel)
The background to this audit is explained in the ‘Background’ section of the report of the Independent Statins Review Panel.
The audit began with a timeline: what had happened when in connection to the two papers. The timeline began with first submission and ran through to the correction of the papers and the publication of the accompanying editorial. These events are now provided in an online timeline on the web page of the Independent Statins Review Panel. Each item is briefly summarised, key issues raised in our discussion are noted, and any outcomes or changed procedures listed. (The ordering of items below reflects the dates along the timeline at which they arose.)
1. Should The BMJ editors have particular procedures for authors who are expert witnesses?
Abramson et al. provided a Conflicts of Interest statement saying that the main author (and another who was added at revision stage) were expert witnesses in statin litigation. Should this have been a cause for concern?
We will continue to ensure that all Conflicts of Interest statements are read by the handling editor. We now empahsise the need for these to be escalated when contentious to a senior colleague; and we will begin to use amended Conflicts of Interest forms that are being developed within the Education section of the journal on all scholarly analysis and commentary articles (see http://www.bmj.com/content/349/bmj.g7197). The issue of expert witnesses continues to be discussed by editors considering conflict of interest issues.
2. Should The BMJ editors insist that authors provide a point by point response to reviewers’ and editors’ concerns?
Abramson et al. did not provide a point by point response to a detailed letter from the editor, which makes it harder to assess how, or how well, the authors have responded.
We will insist on such responses in future.
3. Should The BMJ continue peer review of Observations articles by e-mail, rather than via online manuscript submission and peer review software (ScholarOne)?
As ‘Observations’ are relatively informal articles, they are often submitted via e-mail and may be passed on for peer review in the same format. But this means that reviewers are not guaranteed to receive the standard guidance on reviews, and warnings about confidentiality and conflicts of interest.
We discussed the possibility of including standard texts when reviewing by email but decided instead to insist that when such content is peer-reviewed, it is via the online system in future.
4. Should The BMJ allow peer reviewers to submit comments embedded within a Word or PDF document?
The review submitted by Abramson for the article by Malhotra was sent as comments in a Word document. Arguably these are harder to follow than a narrative reviewers’ report.
We will continue to allow reviewers to submit in this way on occasion, as it may be easier to address concerns to particular parts of the article. Editors should however continue to ensure that they and the authors can understand all points raised by reviewers.
5. Is there a risk of 'cabals', or presenting a one-sided view, if an author of a piece under consideration by the journal reviews another article in the same time frame on the same topic?
Abramson was asked to review Malhotra’s Observations column while his own Analysis article was under consideration.
The editors agree this situation should be avoided in future.
6. The citation of the study by Zhang et al. was added at a late stage in the Abramson article’s development. Is this a problem? Abramson et al. were asked by the editor handling their article to provide numbers and references for their statements about net benefits vs harms and did so.
That there was an error in the numbers and references that were added is unfortunate but not, we feel, related to a process problem at this stage. Technical editors consult handling editors about any late amendments made to articles and will continue to do so. Handling editors escalate issues to more senior colleagues when substantial changes are made.
7. The BMJ editors should resist pressure to publish at a time suited to the general media. authors and their institutions, or other external parties
Extra care should be taken whenever there is a “news agenda” that is not dictated by the journal, or when news cycles drive publication timing.
8. Is there a robust and consistent approach for considering issues raised in Rapid Responses, and for cross-referencing these if the same issue is raised for two articles (as with Abramson et al. and Malhotra)?
A weekly meeting considers all matters arising in Rapid Responses, and the Letters Editor writes to article authors to request responses to substantive Rapid Responses. This occurred for both articles in this instance. New procedures have been put in place to ensure that handling editors are alerted to issues raised in Rapid Responses to articles they have handled, and that article authors are chased up for timely responses to matters arising in this way. In addition, we are committed to making more use of external expert review post-publication if substantive criticisms are raised in rapid responses.
9. What should editors do when complaints about articles are made verbally and/or privately rather than in writing?
Complainants are always encouraged to put complaints in writing and/or to post a Rapid Response on the relevant article if possible, as Professor Collins was on this occasion. However, Professor Collins met with the Editor-in-Chief to discuss his concerns about these two articles in December 2013 and in May 2014. No other editors were present, and Professor Collins did not act on the offer to consider an article from him articulating his concerns.
In future, if complaints are raised in person or by telephone, editors should aim to include relevant colleagues as soon as possible. More than one member of staff should be present at planned meetings and staff should ensure that notes are circulated and approved by all parties. This is particularly important if a complaint is being made about any editor. In general, all article-related matters and all complaints are considered by more than one editor.
10. How can The BMJ avoid delays in articles appearing online, such as those that occurred when the article corrections and associated editorial appeared on bmj.com?
There has been an extensive review of the processes by which articles, and in particular batches of related articles, are put live. We cannot guarantee there will never be delays but are working hard to eliminate any that do occur.
11. Are corrections sufficiently visible?
As is common in scientific journal publishing, it is not routine for corrections to be made to the article PDF. Corrections are, however, very prominent. The corrections appeared on PubMed. after a time lag.
If corrections are particularly important, and if they concern patient safety, they are made to the PDF as well as the HTML version of the article.
In combination with the responses to the suggestions of the Independent panel, the audit concluded that there was little that was substantially exceptional or greatly out of the ordinary about the process of assessing and publishing the two articles in question, but there were things that might be improved about our processes in general to guard against such situations occurring in future.
