Report from BMJ post publication review meeting
Here are the conclusions of our review of our handling of this paper:
The criticisms raised in the rapid responses mainly address the selection and inclusion of studies and the assumptions made by the authors in their modelling analyses. We concluded that these criticisms continue the debate but do not negate the findings of the study.
This article and its accompanying web extras on bmj.com gave an accurate and suitably cautious account of this study's findings, strengths, and limitations. The authors were particularly thorough and transparent in reporting their methods and justifying their assumptions. We noted, too, that the authors had posted an "authors' reply" Rapid Response addressing the criticisms directly (and since the review meeting they have posted another). However, at the review meeting we decided that the following statements in the article:
* at the end of the discussion section: "Coverage of costs by health authorities or health insurers for these preparations and novel prescriptions to patients who have not received other treatments should be discouraged."
* in the abstract: "Health authorities and health insurers should not cover the costs of these preparations, and new prescriptions to patients who have not received treatment should be discouraged"
were not directly supported by their data.
The paper went through detailed peer review and was seen by expert reviewers active in this field. It was also seen by a statistics editor with specific expertise in this type of research methodology, and was accepted at the BMJ's research manuscript committee meeting (subject to revisions, which the authors then made).
At the revision stage the authors responded thoroughly to all comments made by peer reviewers and the BMJ's research editors and statistics editor. One request was for the authors to conduct a standard meta-analysis in addition to their network meta-analysis. This additional analysis was published as web extra 2 alongside the article on bmj.com.
Overall conclusion from post publication review:
The paper was a scholarly and well reported piece of work by highly regarded researchers. The authors' conclusions about funding or prescribing these preparations seemed only indirectly based on their findings and did not add usefully to the article.
The article passed through a rigorous and appropriate peer review process.
As is the case for any scholarly work, it may or may not turn out in good time, and after much debate, to be wrong. We hold the view that science advances by testing, debate, and argument and we consider it unremarkable for some readers to hold different opinions from authors regarding methodology.
Lessons learned and actions we're taking:
We aim to tighten up implementation of our advice on writing structured abstracts for research articles, which reads as follows:
"conclusions - primary conclusions and their implications, suggesting areas for further research if appropriate. Do not go beyond the data in the article. Conclusions are important because this is often the only part that readers look at."
Competing interests: I am senior research editor at the BMJ and chair the weekly research manuscript meeting. Professor Doug Altman, the BMJ's senior statistics editor, declared before the post publication review meeting that he is well acquainted with and has recently been involved in research with two of the article's authors. Hence he did not comment during the review meeting.