Independent statins review panel

Background

In October 2013 The BMJ published an article by John Abramson and colleagues that questioned the evidence behind new proposals to extend the routine use of statins to people at low risk of cardiovascular disease. Abramson et al set out to reanalyse data from the Cholesterol Treatment Trialists (CTT) Collaboration. Their contention was that the benefits of statins in people at low risk were less than has been claimed and the risks greater.

In their conclusion and in a summary box, they said that side effects of statins occur in 18-20% of people. This figure was repeated in another article, by Aseem Malhotra, that was published in The BMJ in the same week. The BMJ and the authors of both articles were made aware that this is incorrect, and corrections were published, withdrawing these statements.

The corrections explain that, although the 18-20% figure was based on statements in the referenced observational study by Zhang et al—which said that "the rate of reported statin related events to statins was nearly 18%,” the articles in The BMJ did not reflect necessary caveats and did not take sufficient account of the uncontrolled nature of the data of Zhang et al.

The BMJ was alerted to the error by Rory Collins, professor of medicine and epidemiology in Oxford and head of the CTT Collaboration, whose data were reanalysed by Abramson et al. Collins called for the retraction of both articles.

As explained in an editorial published alongside the two corrections, The BMJ's editor in chief, Fiona Godlee, decided to pass the decision on whether to retract one or both of the articles to an independent panel. The panel's membership and terms of reference are given below, as is a link to its final report, which was also published as an article in The BMJ.

Independent panel

Independent panel established to review articles published in The BMJ by Abramson et al [1] and Malhotra [2]: Terms of reference

1. Background

In October 2013 The BMJ published an article by John Abramson and colleagues that questioned the evidence behind new proposals to extend the routine use of statins to people at low risk of cardiovascular disease. [1] Abramson et al set out to reanalyse data from the Cholesterol Treatment Trialists Collaboration. Their contention was that the benefits of statins in low risk people were less than has been claimed and the risks greater.

In their conclusion and in a summary box, they said that side effects of statins occur in 18-20% of people. This figure was repeated in another article published in the same week in The BMJ by Aseem Malhotra. [2] The BMJ and the authors of both articles have now been made aware that this is incorrect, and corrections have been published withdrawing these statements.

The corrections explain that, although the 18-20% figure was based on statements in the referenced observational study by Zhang et al --which said that "the rate of reported statin-related events to statins was nearly 18%,” [3] the BMJ articles did not reflect necessary caveats and did not take sufficient account of the uncontrolled nature of Zhang et al's data.

The BMJ was alerted to the error by Rory Collins, professor of medicine and epidemiology in Oxford and head of the CTT collaboration whose data are reanalysed by Abramson et al. Professor Collins has called for the retraction of both articles.

As explained in an editorial, [4] The BMJ's editor in chief Fiona Godlee has decided to pass the decision on whether to retract one or both of the articles to an independent panel, whose membership and terms of reference are given below.

References

1. Abramson JD, Rosenberg HD, Jewell N, Wright JM. Should people at low risk of cardiovascular disease take a statin? BMJ 2013;347:f6123.
2. Malhotra A. Saturated fat is not the major issue. BMJ 2013;347:f6340.
3. Zhang H, Plutzky J, Skentzos S, Morrison F, Mar P, Shubina M, et al. Discontinuation of statins in routine care settings. Ann Intern Med2013;158:526-34.
4. Godlee F. Adverse effects of statins. BMJ 2014; 348:g3306.

2. Membership of the panel

Chair:

Iona Heath, former chair, Royal College of General Practitioners; former chair, The BMJ's ethics committee.

Members:

Stephen Evans, professor of pharmacoepidemiology, London School of Hygiene and Tropical Medicine

Curt Furberg, professor emeritus of public health sciences, Wake Forest University School of Medicine; former chair of steering committee and principal investigator, ALLHAT trial; member, editorial advisory board of The BMJ

Julia Hippisley-Cox, professor of epidemiology and general practice, University of Nottingham

Harlan Krumholz, Harold H. Hines Jr. professor of medicine (cardiology) and professor of investigative medicine and of public health (health policy), Yale School of Medicine

Cynthia Mulrow, senior deputy editor, Annals of Internal Medicine; adjunct professor of medicine, Texas University of Health Science Center; member, Institute of Medicine

Paul Wicks, vice president of innovation, Patients Like Me; member, editorial advisory board of The BMJ

3. Scope of the panel's work

To consider whether either or both articles should be retracted.

To review and comment on the process by which the articles were published.

To review and comment on how criticisms and complaints against the articles were raised, and how the journal responded.

To summarise its findings and make recommendations to The BMJ's editor in chief in a report that will be published on bmj.com.

4. Accountability and transparency

All submissions to the panel and all documents used by the panel will be placed in the public domain on bmj.com, either at the time of submission or as part of the panel's final report.

Panel members and all those submitting information to the panel will make full declarations of their interests, which will be published on bmj.com.

5. Timeframe

To complete review and deliver report by 31 July 2014

To comment on these terms of reference, please post a rapid response or email Jane Smith (jsmith@bmj.com). Consultation period ends Friday 23 May 2014.

The report

The independent panel of internationally renowned experts asked by The BMJ to review its handling of two articles that contained the same error has reported its findings, and were unanimous in their decision that the two papers do not meet any of the criteria for retraction, endorsing The BMJ approach.

After a two-month review, the panel has advised The BMJ that its handling of the two articles was appropriate and that its processes were timely and reasonable.

They have made some recommendations to strengthen our processes, and we will address those. They have also called for the the individual patient data to be made available for independent scrutiny. Patients and their doctors need access to all relevant information to make informed decisions about their health.

Supporting documentation

SP5 Pre publication history for Abramson et al

SP7 and SP8 Post-publication correspondence between The BMJ, Fiona Godlee and Abramson

SP9 Pre-publication history for Malhotra et al

SP9a Attachment to email dated 17 October at 17.34 hrs

SP11 and SP12 Post publication correspondence between The BMJ, Fiona Godlee and Malhotra

SP13 Emails between Rory Collins and Fiona Godlee

SP14 Statins timeline for publication

SP15 Note from Rory Collins for the panel considering the retraction of Abramson and Malhotra papers

SP16 Supplement to note from Rory Collins for the panel considering retraction

SP16a BMJ slides 021213 with notes added subsequently

SP16b Abramson October 2013 with comments final from Collins

SP16c Malhotra October 2013 with comments final from Collins

SP17 Letter 1 from Rory Collins to FG dated 31 March Not for Publication

SP18 Letter 2 from Rory Collins to FG dated 14 April Not for Publication

SP19 Letter 3 from Rory Collins to FG 25 April

SP20 Letter 4 from Rory Collins to FG 28 April Not for Publication

SP21 CTSU grants May 2014

SP22 Initial response to BMJ panel May 20 2014 Abramson et al

SP23 Response to BMJ panel by Abramson et al (June 7, 2014) FINAL

SP24 Response from Malhotra to Rory Collins submission

SP25 Submissions to Panel from 13 individuals or groups

SP26 Declarations of interest by the authors of submissions

SP27 List of documents accompanying the report

SP28 Declarations of interests of panel members

SP30 Proforma for Statistical review of Abramson et al

SP31 Statistical report on Abramson by Stephen Evans

SP32 Statistical report on Abramson et al by Carol Coupland

Response from The BMJ

Several recommendations were made by the independent panel that was convened to consider the question of retraction, and to review the processing, of the articles by Abramson and Malhotra. This document itemises the journal's response to those recommendations. Further details of each suggestion from the panel and its context can be found in the panel’s final report; page numbers for the report are given after each item below, and items are considered in the order in which they occur in the report.

1. Initial Submission (p11)

“Editors could consider developing guidance for internal use in relation to when Analysis articles that contain calculations and numerical extrapolations warrant statistical review.”

Such guidelines have been developed, and statistical reviews analogous to those provided for research articles are now sought for Analysis articles that rely on statistical argument.

2. Revision and Publication: Summary and Assessment (p12)

• “Senior or deputy editors should sign off revisions that include substantive additions to an article or changes in a “fact” box.”
  
  Such changes are always seen by the editor handling the article in addition to the technical editor who is copy-editing it. In addition, handling editors are encouraged to escalate any concerns to the relevant Deputy Editor.

• “Extra attention should be given to manuscripts that have been noted by reviewers and editors to be controversial and potentially slanted or one-sided. “
  
  Additional flags and warnings are now put in place for particularly controversial articles, both on the manuscript tracking system and via discussion at team meetings.

• “Editors might consider whether such pieces are best placed (and clearly labelled) as opinion pieces or whether they are best placed and clearly labelled as articles/analyses that are meant to have a more comprehensive and, according to author instructions, “an even-handed approach in evaluating evidence, a lucid line of argument, and a worthwhile conclusion.”
  
  The editors feel that there is a place for articles that combine data and argument to make a case, and that Analysis is the format currently identified for articles of this type.
  

• “Although the article by Abramson et al was not press released, that by Malhotra, which was also clearly presenting a controversial view, was ( see press release). Press releases should be used with great care in such contexts.”

  Press releases are always seen by the authors of the article in question and by a Deputy Editor acting as ‘Duty Editor’. A weekly planning meeting ensures that particularly controversial articles are flagged up to the press release team. Our press releases aim to give a straightforward summary of the views as set out in the article. Because the press release can only provide a very brief summary of a paper, we always provide journalists with the full text of articles and author contact details

• “Editors might give extra attention to the following issues when considering potentially controversial articles that postulate strong arguments: possible selective citing of material, failure to critically appraise evidence that is used to support authors’ arguments, and over criticism of evidence that does not support authors’ arguments.“
  
  Editors are now alerted to consider these issues in particular in controversial articles, and to ask reviewers to do so.

• “Editors should carefully consider whether articles that include extrapolations and recalculations of numerical data need statistical review. “
  

  As noted in response to item 1, we do now seek specific statistical advice for such articles.

• “They should also consider whether a paper that has been revised requires additional peer review.”
  
  Editors discussed whether there should be a blanket policy of re-review after revision and decided against. However, re-review is required when substantial additions have been made, or changes to data and analysis, or when specific details have been requested by a specific reviewer, or after other  escalation  with colleagues. Editors should err on the side of caution when considering the need for re-review.

3. Review and comment on how criticisms and complaints against the articles were raised, and how the journal responded (p 14)

“The BMJ editorial staff should implement a significant event audit in relation to the need for the correction. The aim of the audit would be to try and identify what would need to have been in place to ensure that the correction was made in a more timely fashion.”

The editors undertook an end-to-end critical event audit of the handling of the two papers and their associated corrections. This is presented in a separate document.

 

BMJ critical event audit: outcomes

The independent Statins Review that was convened to consider issues around the publication of two articles, by Abramson et al. and Malhotra, recommended that “The BMJ editorial staff should implement a significant event audit in relation to the need for the correction. The aim of the audit would be to try and identify what would need to have been in place to ensure that the correction was made in a more timely fashion.” The editors undertook an end-to-end critical event audit of the handling of the two papers and their associated corrections. This document describes this e BMJ’s audit and its recommendations.

This audit was completed by current and former senior editorial staff of the journal, with oversight of the Editor-in-Chief, Fiona Godlee. The following staff contributed:

• Theodora Bloom, Executive Editor (since June 2014)
• Tony Delamothe, Deputy Editor
• Fiona Godlee, Editor-in-Chief
• Trevor Jackson, Head of News and Comment
• Karl Sharrock, Lead Technical Editor, News and Comment
• Jane Smith (Deputy Editor until end 2012 and Secretary to the Independent Panel)

The background to this audit is explained in the ‘Background’ section of the report of the  Independent Statins Review Panel.

The audit began with a timeline: what had happened when in connection to the two papers. The timeline began with first submission and ran through to the correction of the papers and the publication of the accompanying editorial. These events are now provided in an online timeline on the web page of the Independent Statins Review Panel.  Each item is briefly summarised, key issues raised in our discussion are noted, and any outcomes or changed procedures listed. (The ordering of items below reflects the dates along the timeline at which they arose.)

1. Should The BMJ editors have particular procedures for authors who are expert witnesses?

Abramson et al. provided a Conflicts of Interest statement saying that the main author (and another who was added at revision stage) were expert witnesses in statin litigation. Should this have been a cause for concern?

We will continue to ensure that all Conflicts of Interest statements are read by the handling editor. We now empahsise the need for these to be escalated when contentious to a senior colleague; and we will begin to use amended Conflicts of Interest forms that are being developed within the Education section of the journal on all scholarly analysis and commentary articles (see http://www.bmj.com/content/349/bmj.g7197). The issue of expert witnesses continues to be discussed by editors considering conflict of interest issues.  

2. Should The BMJ editors  insist that authors provide a point by point response to reviewers’ and editors’ concerns?

Abramson et al. did not provide a point by point response to a detailed letter from the editor, which makes it harder to assess how, or how well, the authors have responded.

We will insist on such responses in future.

3. Should The BMJ continue peer review of Observations articles by e-mail, rather than via online manuscript submission and peer review software (ScholarOne)?

As ‘Observations’ are relatively informal articles, they are often submitted via e-mail and may be passed on for peer review in the same format. But this means that reviewers are not guaranteed to receive the standard guidance on reviews, and warnings about confidentiality and conflicts of interest.

We discussed the possibility of including standard texts when reviewing by email but decided instead to insist that when such content is peer-reviewed, it is  via the online system in future.

4. Should The BMJ allow peer reviewers to submit comments embedded within a Word or PDF document?

The review submitted by Abramson for the article by Malhotra was sent as comments in a Word document. Arguably these are harder to follow than a narrative reviewers’ report.

We will continue to allow reviewers to submit in this way on occasion, as it may be easier to address concerns to particular parts of the article. Editors should however continue to ensure that they and the authors can understand all points raised by reviewers.

5. Is there a risk  of 'cabals', or presenting a one-sided view, if an author of a piece under consideration by the journal reviews another article in the same time frame on the same topic?

Abramson was asked to review Malhotra’s Observations column while his own Analysis article was under consideration.

The editors agree this situation should be avoided in future.

6. The citation of the study by Zhang et al. was added at a late stage in the Abramson article’s development. Is this a problem? Abramson et al. were asked by the editor handling their article to provide numbers and references for their statements about net benefits vs harms and did so.

That there was an error in the numbers and references that were added is unfortunate but not, we feel, related to a process problem at this stage. Technical editors consult handling editors about any late amendments made to articles and will continue to do so. Handling editors escalate issues to more senior colleagues when substantial changes are made.

7. The BMJ editors should resist pressure to publish at a time suited to the general media. authors and their institutions, or other external parties

Extra care should be taken whenever there is a “news agenda” that is not dictated by the journal, or when news cycles drive publication timing.

8. Is there a robust and consistent approach for considering issues raised in Rapid Responses, and for cross-referencing these if the same issue is raised for two articles (as with  Abramson et al. and Malhotra)?

A weekly meeting considers all matters arising in Rapid Responses, and the Letters Editor writes to article authors to request responses to substantive Rapid Responses. This occurred for both articles in this instance. New procedures have been put in place to ensure that handling editors are alerted to issues raised in Rapid Responses to articles they have handled, and that article authors are chased up for timely responses to matters arising in this way. In addition, we are committed to making more use of external expert review post-publication if substantive criticisms are raised in rapid responses.

9. What should editors do when complaints about articles are made verbally and/or privately rather than in writing?

Complainants are always encouraged to put complaints in writing and/or to post a Rapid Response on the relevant article if possible, as Professor Collins was on this occasion. However, Professor Collins met with the Editor-in-Chief to discuss his concerns about these two articles in December 2013 and in May 2014. No other editors were present, and Professor Collins did not act on the offer to consider an article from him articulating his concerns.

In future, if complaints are raised in person or by telephone, editors should aim to include relevant colleagues as soon as possible. More than one member of staff should be present at planned meetings and staff should ensure that notes are circulated and approved by all parties. This is particularly important if a complaint is being made about any editor. In general, all article-related matters and all complaints are considered by more than one editor.

10. How can The BMJ avoid delays in articles appearing online, such as those that occurred when the article corrections and associated editorial appeared on bmj.com?

There has been an extensive review of the processes by which articles, and in particular batches of related articles, are put live. We cannot guarantee there will never be delays but are working hard to eliminate any that do occur.

11. Are corrections sufficiently visible?

As is common in scientific journal publishing, it is not routine for corrections to be made to the article PDF. Corrections are, however, very prominent. The corrections appeared on PubMed. after a time lag.

If corrections are particularly important, and if they concern patient safety, they are made to the PDF as well as the HTML version of the article.

In combination with the responses to the suggestions of the Independent panel, the audit concluded that there was little that was substantially exceptional or greatly out of the ordinary about the process of assessing and publishing the two articles in question, but there were things that might be improved about our processes in general to guard against such situations occurring in future.