Higgins et al should justify the inclusion of the study from Papua New Guinea
On 23rd February, I formally complained about the peer review and editorial process leading to the publication of Higgins et al (BMJ 2016;355:i5170) . Both the BMJ  and Higgins et al  gave unsatisfactory replies to my complaint, and have not responded to my criticism of their replies .
During the deliberations of the WHO Working Group on the Non-Specific Effects of Vaccines, many "commentators" made suggestions that led to changes in the initial classifications circulated by Higgins et al, so the published review is already "post hoc" by their definition. I can provide evidence to support this claim if it is disputed by Higgins et al.
During and after the review process Higgins et al, with support from the BMJ, have repeatedly refused to justify their decision to include the study from Papua New Guinea (PNG) in their assessment of DTP on the grounds that any change would be “post hoc”. Higgins et al admit that the decision to include studies is subjective, but imply that these subjective decisions should be made only by them, “based on information presented by study authors rather than from commentators (including peer reviewers).”
It is misleading for Higgins et al to imply that they developed objective criteria to evaluate studies, applied these criteria in a dispassionate manner based only on information presented by the study authors, then made no subsequent alterations because such changes would be “post hoc”. In fact, Higgins et al changed the selection of studies in response to suggestions made during the WHO Working Group deliberations.
First, Higgins et al should be required to provide a detailed response to each of the four reasons why the PNG study should have been excluded, as presented in my post of 9th March . The specious argument that any change would be “post hoc” should not be accepted. Second, unless the authors can conclusively rebut all four criticisms, a correction should be published stating that the systematic review gives a relative risk for DTP of 1.53 (95% CI 1.02-2.30), replacing the estimate of 1.38 (0.92-2.08) which includes the PNG study.
I am sorry to be so critical. In fact, BMJ has made a major contribution by publishing this important systematic review (and many other papers relating to health in low-income countries). I am a fervent supporter of vaccines – indeed I played a major role in developing the pigbel vaccine that was used in the PNG study (Lancet 1979;313:227-230).
1. Shann F. Poor science or fraud by BMJ staff. http://www.bmj.com/content/355/bmj.i5170/rr-6.
2. Weber WE, Loder E, Riley R. Re: Association of BCG, DTP, and measles containing vaccines with childhood mortality: systematic review. http://www.bmj.com/content/355/bmj.i5170/rr-8.
3. Higgins JPT, Soares-Weiser K, Sterne JAC, Reingold AL. Re: Association of BCG, DTP, and measles containing vaccines with childhood mortality: systematic review. http://www.bmj.com/content/355/bmj.i5170/rr-7.
4. Shann F. A flawed "post hoc" estimate or a more accurate "post post hoc" estimate? http://www.bmj.com/content/355/bmj.i5170/rr-9.
Competing interests: No competing interests