Re: Association of BCG, DTP, and measles containing vaccines with childhood mortality: systematic review
Dear Dr. Shann,
Thank you for your letter complaining about the peer review and editorial process leading to the publication of Higgins et al, BMJ 2016;355:i5170.
Our answer to your two complaints is as follows:
1. Papua New Guinea study included to avoid a “dramatic effect” of DTP.
Your complaint here is that we recommended to keep this study included in the meta-analysis “to avoid a dramatic effect on the final estimate”. You find this problematic, as this would be a post-hoc decision: you correctly argue that “a study should be included or excluded solely on the basis of the attributes of the study, and not the effect on the final estimate”.
We are afraid that you misunderstood our recommendations to the authors in the revision process. Our recommendation to the authors was to stick with the analysis as they originally planned to do: to exclude studies deemed ‘very high risk of bias’ and to keep studies with ‘high risk of bias’ in the analysis. Our first argument is that exclusion would be a post hoc decision (So in line with your reasoning). The problem with this analysis was that all studies had considerable risks of biases and at a certain point one has to draw a line between studies to include and to exclude. The authors had a priori formulated their own risk of bias criteria and we preferred that they stick with these.
As a secondary consequence we observed that exclusion of that study would increase significance, with CI now excluding 1. This was not the main reason for us to recommend inclusion, but to us this looked as confirmation that this would be the safer approach in a meta-analysis with 10 studies which are all deemed at high risk of bias.
The authors did include a meta-analysis without the PNG study, see:
“The mortality rate was very high among unvaccinated children in the Papua New Guinea study,18 and two referees had notable concerns about this study. Excluding it from the meta-analysis gave a relative risk of 1.36 (1.09 to 1.68).”
We do acknowledge that in this analysis, the selection of studies was ultimately subjective. We also feel that publication of the peer reviews and editorial communications does make these processes more transparent, so that the reader can judge these for her/him self.
2. The wrong summary measure was used after removing the PNG study - without peer review
You are correct, and the authors have issued a response explaining their error:
We hope that you will find these answers satisfactory
Competing interests: No competing interests