A flawed “post hoc” estimate or a more accurate “post post hoc” estimate?
Both Weber et al  and Higgins et al  concede that the selection of studies is subjective, but then claim that every one of the subjective selections made by Higgins et al was optimal, and reject the strong recommendation of two of the three reviewers for exclusion of the study from Papua New Guinea (PNG) .
Despite their claims, the record shows that Weber et al considered how the DTP estimate would be affected by exclusion of the PNG study. They initially suggested exclusion on the grounds that this would reduce the RR for DTP from 1.38 to 1.36, but later reversed this because “excluding the study is post hoc and gives a more dramatic effect (as it increases significance, with CI now excluding 1).” Peer review is an important part of the process of scientific publication, so changes made during review are not “post hoc” (after the event).
Higgins et al imply that the subjective decisions about study selection should be made only by them, and “based on information presented by the study authors rather than from commentators (including peer reviewers).” In fact, during the deliberations of the WHO Working Group on the Non-Specific Effects of Vaccines, many “commentators” made suggestions that led to changes in the initial classifications circulated by Higgins et al, so the published review is already “post hoc” by their definition. Higgins et al give no justification for ignoring facts from sources other than the original authors but, as it happens, the salient facts about the PNG study are provided by the authors. At no stage have Higgins et al answered the reviewers’ specific criticisms of the PNG study.
There are at least four reasons why the PNG study should have been excluded because of very high risk of bias. First, pigbel vaccine was given with DTP, and the criteria mandate exclusion of studies if “any other vaccine [is] given with any dose of DTP (other than OPV)” . This alone is an unequivocal objective reason for excluding the PNG study.
Second, there was extreme frailty bias because vaccines were given only at clinics, and clinics were instructed not to vaccinate sick children (p.147 of the PNG paper); if they were sick, unvaccinated children were not given BCG, and BCG-vaccinated children were not given DTP. The region had a 1-5 month mortality rate of 33 per 1000 live births . In children aged 1-5 months in the study, mortality was an extraordinary 233 per 1000 person-years in unvaccinated children compared to only 31 per 1000 person-years in vaccinated children (table 3); this is very strong evidence of bias .
Third, retrospective updating was used and no child aged 1-5 months was excluded for lack of information. If the data are not perfect this results in survival bias (when a dead child who is classified as unvaccinated or BCG-vaccinated has actually received DTP) . The PNG authors admit that their DTP data were unreliable (page 147): when a child died, they carefully checked the polysaccharide pneumococcal vaccine status (to get accurate information for a randomised trial), but they did not check the DTP status.
Fourth, the systematic review is testing the effect of giving DTP after BCG. However, 24% of children received BCG with or after DTP (table 3 of the PNG paper), and this reduces the harm from DTP (Higgins et al, Figure 5 third analysis) so the PNG data underestimate harm from DTP.
What is the point of peer review if reviewers' suggestions can be dismissed as “post hoc” or “from commentators (including peer reviewers)” without detailed refutation? Such an approach undermines an important safeguard in the scientific process. Unless the authors can conclusively rebut all four criticisms, a correction should be published stating that the systematic review gives a DTP effect of 1.53 (95% CI 1.02-2.30).
Professorial Fellow, Department of Paediatrics, University of Melbourne
1. Weber WM, Loder E, Riley R. Re: Association of BCG, DTP, and measles containing vaccines with childhood mortality: systematic review. http://www.bmj.com/content/355/bmj.i5170/rr-8
2. Higgins JPT, Soares-Weiser K, Sterne JAC, Reingold AL. Re: Association of BCG, DTP, and measles containing vaccines with childhood mortality: systematic review. http://www.bmj.com/content/355/bmj.i5170/rr-7
3. Lehmann D, Vail J, Firth MJ, et al. Benefits of routine immunizations on childhood survival in Tari, Southern Highlands Province, Papua New Guinea. Int J Epidemiol 2005;34:138-48.
4. Fine PEM, Smith PG. ‘Non-specific effects of vaccines’- an important analytical insight, and call for a workshop. Trop Med Int Health 2007;12:1–4.
5. Lehmann D, Vail J, Vail P et al. Demographic Surveillance in Tari, Southern Highlands Province, Papua New Guinea: Methodology and Trends in Fertility and Mortality between 1979 and 1993. Goroka and Tari: Papua New Guinea Institute of Medical Research, 1997: Table 8.
6. Aaby P, Ravn H, Benn CS. The WHO Review of the Possible Non-Specific Effects of Diphtheria-Tetanus-Pertussis Vaccine. Pediatr Infect Dis J 2016;35:1247-57.
Competing interests: No competing interests