Covid-19: Pfizer BioNTech vaccine reduced cases by 94% in Israel, shows peer reviewed study
BMJ 2021; 372 doi: https://doi.org/10.1136/bmj.n567 (Published 25 February 2021) Cite this as: BMJ 2021;372:n567Read our latest coverage of the coronavirus outbreak
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Dear Editor,
This note aims to raise serious concerns regarding upward bias in the studies conducted in Israel to assess the efficacy of the BNT162b2 mRNA Covid-19 vaccine by Pfizer.
Multiple studies were conducted thus far to assess the efficacy of the Pfizer’s COVID-19 vaccine. As part of the phase 3 clinical trials, Pfizer conducted a placebo-controlled, observer-blinded trial. Polack et al. (f1) (2020, hereafter Polack) report the finding of this trial. They conclude that the vaccine has 95% efficacy, relying on the respective numbers of positive PCR tests, 162 in the control (placebo) group versus 8 in the treatment (vaccinated) group. Doshi (2021) (f2) raised serious concerns about the validity of the conclusion of “95% efficacy”. The concerns point to the fact that in the FDA report on the Pfizer’s vaccine, there is a category of “suspected Covid-19”—those with Covid-19-like symptoms that were not confirmed by a PCR test. In this category, the split between vaccinated and unvaccinated is roughly equal, 1594 vaccinated versus 1816 unvaccinated. Another concerning fact according to Doshi is that many more vaccinated were excluded from the trial than placebo (311 versus 60).
These concerns, among others, emphasize the importance of the large-scale observational studies. Indeed, the massive and early vaccination campaign in Israel provides such opportunity and two related studies were conducted, Dagan et al. (2021, hereafter “Dagan”) (f3) and Haas et al. (2021, hereafter “Haas”) (f4). The latter is based on the entire Israeli population and the former on about half of the Israeli population (members of the largest HMO, Clalit Health Services) and also has a shorter study period. Notably, Dagan employs a different methodology from Hass for matching patients between the vaccinated and unvaccinated groups. Wise (2021) (f5) highlights the main finding of Dagan and their implications.
These two studies estimate the vaccine efficacy against symptomatic or asymptomatic infection (confirmed by positive PCR test) as 92% (Dagan) and 95% (Haas), starting seven days after the second dose.
Unfortunately, both papers suffer from a potential serious upward bias in their estimates, particularly with respect to asymptomatic infection and potentially also with respect to symptomatic infection with no hospitalization. The concerns stem from the fact that Israel has been mandating strict isolation on anyone who was exposed to an infected individual. The isolation is mandated to 14 days but can be shortened to only 10 days with two negative PCR tests. In contrast, vaccinated individuals have been exempted from any isolation requirement whatsoever, starting seven days after the second dose. The isolation regulations have been enforced quite strictly in Israel. In particular, during the study period in Haas there were 430,000 isolations (f6). Thus, there is a serious concern that the testing intensity of vaccinated individuals seven days after the second dose will be significantly lower compared to unvaccinated individuals. Notably, Dagan does not refer to this potential source of bias. Haas refers to the concern but mentions that only 19% of the 4.4 million PCR tests in the relevant study period were done on exempted (vaccinated) people, a fact that supports the concern above that significantly more tests were conducted among unvaccinated individuals.
The concerns regarding the potential upward bias in the efficacy of the vaccine as a result of different testing intensity are supported by additional findings. Dagan reports in Table 2 only 60% for the first seven days after the second dose with a significant increase to 92% beyond seven days after the second dose, which is exactly when testing exemption would have started. In contrast, in the Pfizer experiment, (see Figure 3 in Polack et al.) the respective increase in efficacy between the first seven days after the second dose and the beyond seven days is very small (from 90.5% to 95%). Moreover, Figure 3 in Polack shows that the vaccine reaches almost full efficacy (which is 95%) before day 14 after the first dose, which again is seemingly contradictory to the dramatic jump in the estimates of Dagan.
For some unknown reasons, Haas does not report the assessed efficacy for the first seven days after the second dose, and only reports the efficacy beyond seven days. However, in the appendix they do provide efficacy estimates for the period of 14-21 days after the first dose (appendix p. 5). Interestingly, the efficacy estimate for this period of time is only 58%, similar to the 60% estimate above by Dagan.
In conclusions, there are serious concerns, that the estimates of Dagan and Haas regarding the efficacy of the Pfizer vaccine against infection for the period from seven days after the second dose both suffer from significant upward bias. The suspected bias stems from lack of appropriate control in their analyses for the apparent differences in testing intensity between unvaccinated individuals and vaccinated individuals, particularly starting seven days after the second dose. Detail data about the tests conducted in the two populations during the respective periods studied could potentially allow reducing the bias of the estimates.
*Levi and Wohl contributed equally to this response
**We thank Ehud Qimron for helpful comments and suggestions
Footnotes
1 Polack FP, Thomas SJ, Kitchin N, et al. Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine. N Engl J Med 2020; 383: 2603-15.
2 Doshi P., Pfizer and Moderna’s “95% effective” vaccines—we need more details and the raw data, British Medical Journal 2021, available online from January-4-2021 in the Opinions section.
3 Dagan N, Barda M, Kepten E, et al. BNT162b2 mRNA COVID-19 vaccine in a nationwide mass vaccination setting. N Engl J Med 2021; 384: 1412–23.
4 Hass, EJ, Angulo, FJ.McLaughlin, JM. Et al. Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalizations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data, The Lancet, 2021, available online from May-5-2021.
5 Wise, J., Covid-19: Pfizer BioNTech vaccine reduced cases by 94% in Israel, shows peer reviewed study, BMJ 2021;372:n567.
6 According to the Israeli Ministry of Health’s data. See https://data.gov.il/dataset/covid-19 .
Competing interests: No competing interests
Dear Editor,
So far Pfizer BioNTech’s vaccine has been 94% effective in reducing Covid-19 cases during Israel’s mass vaccination campaign. (Wise, BMJ 2021;372:n567, Feb 25) This refers to relative population benefit, but many people want to know about individual benefit, which can be expressed as absolute risk reduction (ARR). The following table provides these numbers, based on Figure 2 in the original article by Dagan et al, NEJM 2021, online Feb 24, DOI:10.1056/NEJMoa2101765. There were 596,618 subjects in each group.
TOTAL EVENTS BY 42 DAYS AFTER FIRST DOSE OF COVID-19 VACCINE
PCR+ Symptomatic Hospitalized Severe illness Death
Unvaccinated 6100 3607 259 174 32
Vaccinated 4460 2389 110 55 9
ARR 0.0027 0.002 0.00025 0.0002 0.000039
NNTV 370 500 4000 5000 25,641
NNTV is the Number Need To Vaccinate, and = 1/ARR. For example, 5000 individuals must be vaccinated to prevent one severe illness from Covid-19. The other 4999 individuals derive no benefit but are subject to adverse vaccine effects, which are yet to be quantified. Numbers will change as time goes on and new data are acquired.
Public health authorities in the US continue to push for universal vaccination and continued lockdowns, even as cases rapidly decline. However, some experts believe we need to open things up (Makary, “We’ll Have Herd Immunity by April”, Wall Street Journal, 19 February 2020) Some of us believe that vaccination should be targeted at the ~20% of the population that is truly high risk, and let the rest of the population acquire broader and more lasting protection from natural infection, an approach that may be safer and more cost-effective overall. (https://www.bmj.com/content/372/bmj.n393/rr-2)
ALLAN S. CUNNINGHAM 28 February 2021
Competing interests: No competing interests
A cohort study of the COVID-19 vaccine should clarify the effects of the "premature efficacy bias"
Dear Editor
Since they were granted emergency use authorization, COVID-19 vaccinations have been carried out on a large scale across the world. However, no vaccine has yet been officially approved, and RCTs have left many unanswered questions about the nature of their effectiveness.(f1) For example, regulators made clear that RCTs did not robustly answer questions regarding the prevention of asymptomatic infection, transmission, hospitalization, and death.(f2)
Beyond RCTs, cohort studies from countries with large vaccinations have been published. Some data on the effectiveness of the Israeli vaccine was published in correspondence by Amit et al,.(f3) In addition, a large-scale study by Dagan et al. reported that vaccination reduced SARS-CoV-2 infections, symptomatic COVID-19, hospitalization, severity and mortality. (f4) In Scotland, Vasileiou et al. reported that the COVID-19 vaccine reduced hospitalizations from SARS-CoV-2 infections.(f5) Hall et al. reported a study of health care workers in England that reduced the number of infected people, including asymptomatic.(f6)
All of these studies, however, are vulnerable to the "premature efficacy" bias which can be seen within two weeks after vaccination in all of these studies, a period of time in which we should see little to no efficacy.
Regarding this issue, dubbing it “early effects”, Vasileiou wrote four “Possible explanations for these early effects”. But the researchers discarded the effect of them in their final analysis.
Hall et al. explain this "premature efficacy" issue as occurring amongst those "who are symptomatic, currently PCR positive, or have been recently exposed to a COVID-19 case might defer their vaccination and be under-represented in accordance with national guidance." A report from Israel also reveals a "premature efficacy" effect, but the authors do not explain it.(f3)(f4)
This issue was also considered at a WHO meeting and described as follows: "Several biases can occur in the first couple of weeks after vaccination.”(f7) However, it does not mention here how this problem affects efficacy estimates after the period of "premature efficacy". When inoculated before an epidemic begins, such as with influenza vaccine, it does not appear in a clear form like COVID-19 vaccine. However, in cohort studies of influenza vaccine, "healthy vaccinee bias" is inherent.(f8)
I compared the time and course of the "premature efficacy" and its strength, the number of days since vaccination and the course thereafter, between the reports. To compare the data between reports, all data were limited to Pfizer-BioNTech vaccine, and only the data of the initial vaccination dose were used. The unit of the original document was used. In the report by Hall et al., hazard ratios are not shown, so the numbers were hand measured from Fig. 2B. The report by Dagan N et al. calculated the relative risk.
When the outcome was "infection," as in the case of Hall et al., an "effect" with a hazard ratio of 0.54 appeared 0-3 days after vaccination, and the effect decreased to 0.95 on 4-6 days. After that, it increased to 0.91 on 7-9 days, 0.77 on 10-13 days, 0.45 on 14-20 days, and 0.32 after 21 days. Even for "infections" with the same outcome, Amit et al. reported a '1-rate reduction' of 0.7 on 0-14 days and 0.25 on 15-28 days after vaccination. However, Dagan et al. reported a relative risk of 0.83 on 0-7 days after vaccination and 0.97 with almost no "effect" at 8-14 days. After that, it was 0.52 on 15-21 days and 0.41 on 22-28 days. The degree of effect in the early post-vaccination period between reports is significantly different, and the time and degree of subsequent effect is also quite different.
For hospitalizations, Vasileiou et al. reported a very large "early effect" with a post-vaccination RR of 0.14 on days 0-6 and 0.47 on days 7-13. After 14 days, the effect increased to 0.31, 0.22 on 35-41 days. On the other hand, Dagan et al. reported the RR was 0.46 from 0 to 7 days after vaccination, but the "effect" decreased rapidly to 0.87 from 8 to 14 days. After that, it was 0.46 on 15-21 days, 0.83 on 22-28 days, and 0.66 on 29-35 days, and the effects of these periods are significantly different from those reported by Vasileiou et al.
The fact that the "premature efficacy" varies in degree and duration according to reports may be caused by many factors. The "premature efficacy" bias occurs " in the first few days after the first dose,"(f7) in “Individuals who are experiencing early COVID-19 symptoms or who were recently exposed might defer vaccination," It is difficult to say, and it is considered to be related to time-independent bias such as "healthy vaccinee bias" which also affects the results after that. Therefore, I consider that the exact effect of the vaccine cannot be estimated unless at least the effect of this “premature efficacy” bias is corrected in some way.
The above is the data only for the first vaccination, but it is clear from Fig. 2 of paper by Hall et al that the same bias occurs at the time of the second vaccination.
*I thank Peter Doshi for helpful comments and suggestions.
Footnotes
1 https://blogs.bmj.com/bmj/2021/01/04/peter-doshi-pfizer-and-modernas-95-...
2 https://www.fda.gov/media/144245/download#page=42
3 Amit S, Regev-Yochay G, Afek A, et al. Early rate reductions of SARS-CoV-2 infection and COVID-19 in BNT162b2 vaccine recipients. Lancet. 2021; 397: 875–7.
4 Dagan N, Barda N, Kepten E et al. BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Mass Vaccination Setting. N Engl J Med 2021;384:1412-23.
5 Vasileiou E, Simpson CR, Shi T, et al. Interim findings from first-dose mass COVID-19 vaccination roll-out and COVID-19 hospital admissions in Scotland: a national prospective cohort study. Lancet.2021 May 1; 397:1646–57.
6 Hall VJ, Foulkes S, Saei A, et al. COVID-19 vaccine coverage in health-care workers in England and effectiveness of BNT162b2 mRNA vaccine against infection (SIREN): a prospective, multicentre, cohort study. Lancet.2021;397:1725–35.
7 Patel MK, Bergeri I, Bresee JS, et al. Evaluation of post-introduction COVID-19 vaccine effectiveness: Summary of interim guidance of the World Health Organization. Vaccine. 2021 doi: 10.1016/j.vaccine.2021.05.099 [Epub ahead of print]
8 Remschmidt C, Wichmann O, Harder T. Frequency and impact of confounding by indication and healthy vaccinee bias in observational studies assessing influenza vaccine effectiveness: a systematic review. BMC Infect Dis.2015;15:429.
Competing interests：No competing interests
12 July 2021/07/14
Keiji Hayashi (Medical Doctor, Hayashi Children’s Clinic) hayashi@kch.biglobe.ne.jp
Competing interests: No competing interests