In a commissioned commentary, Chris Baraniuk reviews the “knowns and unknowns” about Russian vaccines against Covid-19, with a specific focus on Sputnik V [1]. While the commentary correctly emphasizes the inconsistencies identified in the phase 1/2 trial results published in the Lancet [2], it mainly discusses the more recently published phase 3 trial results [3].
Our previous concerns regarding the phase 1/2 trial included problematic data patterns with an excess homogeneity of vaccine efficacy across different time points [4]. The authors responded that the unusual data pattern was “a coincidence” due to the small sample size of their study and the discrete distributions of their outcomes [5].
Following such a reasoning, inconsistencies should not be expected in the subsequent larger phase 3 trial. However, we noticed an unexpected homogeneity of vaccine efficacy, this time between age groups. This analysis is central in the Lancet paper at issue because of the disproportionate disease burden in older people. Of course, implausible results can still be observed by chance. However, we have also identified a similar feature, i.e. an excessive homogeneity of the reported vaccine efficacy in the values reported in earlier interim analyses and the published article.
On 11 November 2020, a first press release announced a 92 % efficacy [6]. From this press release we can compute that there were four Covid cases in the vaccine group and 16 in the placebo group. On 24 November 2020, a second press release announced a 91% efficacy with 8/14,095 cases in the vaccine group and 31/4,699 in the placebo group [7]. On 14 December 2020 a third press release announced again a 91% efficacy with 16/17,032 cases in the vaccine group and 62/5,682 in the placebo group [8]. Much to our surprise, the number of cases was exactly twice the number of cases observed in the previous press release [7] in both the placebo and the vaccine groups and about fourfold the number of cases in the vaccine and placebo groups in the first press release [6]. Lastly, the Lancet publication, with a database lock on 24/11/2020, announced a 92% efficacy with 16/14,964 cases in the vaccine group and 62/4,902 in the placebo group [3]. Altogether these results again appear “too good to be true”. The ratio of the number of events between the placebo and vaccine groups is as homogeneous as can be permitted by the number of events (20, 39 and 78) in the press releases [6-8].
The bootstrap P-value for assessing homogeneity of vaccine efficacy between interim stages by means of Pearson’s goodness-of-fit statistic was .9864, taking into account that the total number of events per interim stage is fixed. The Breslow-Day P-value for assessing the homogeneity of vaccine efficacy in age subgroups is equal to .9963. This means that vaccine efficacies are excessively similar between interim analyses and age subgroups even if we were to assume perfect homogeneity. The unusual and improbable high homogeneity of the vaccine efficacy across age strata and different interim analyses raises concerns about the data reported.
In a recent letter to The Lancet documenting these concerns (this letter was submitted upon invitation by The Lancet), the editorial team agreed to publish the letter on condition that we removed our concerns on the press release results since these results were not part of the Lancet publication. We, however felt it important to forward our concerns to the European Medicine Agency which we did on 03/12/2021. The recent BMJ publication by Baraniuk C [1] highlights the phase 3 data but did not address these concerns. We therefore felt it pertinent to inform readers of also these concerns with the Sputnik V vaccine data.
Enrico M. Bucci (Sbarro Institute - Temple University Department of Biology, USA), Johannes Berkhof (Amsterdam University Medical Centers, Department of Epidemiology and Data Science, Vrije Universiteit Amsterdam, Netherlands), André Gillibert (Department of Biostatistics, CHU Rouen, France), Gowri Gopalakrishna (Amsterdam University Medical Centers, Department of Epidemiology and Data Science, Vrije Universiteit Amsterdam, Netherlands), Raffaele A Calogero (Department of Molecular Biotechnology and Health Sciences, University of Torino, Italy), Anders Bjorkman (Department of Microbiology, Tumor and Cell Biology, Karolinska Institutet, Stockholm, Sweden), Lex M. Bouter (Amsterdam University Medical Centers, Department of Epidemiology and Data Science, Vrije Universiteit Amsterdam, Faculty of Humanities, Department of Philosophy, Netherlands), Konstantin Andreev (Howard Hughes Medical Institute, Department of Molecular Biosciences, Northwestern University, Evanston, USA), Florian Naudet (Univ Rennes, CHU Rennes, Inserm, CIC 1414 (Centre d’Investigation Clinique de Rennes), France), Vasiliy Vlassov (HSE University, Moscow, Russia)
REFERENCES
1. Baraniuk C. Covid-19: What do we know about Sputnik V and other Russian vaccines? BMJ (Clinical research ed) 2021;372:n743. doi: 10.1136/bmj.n743 [published Online First: 2021/03/21]
2. Logunov DY, Dolzhikova IV, Zubkova OV, et al. Safety and immunogenicity of an rAd26 and rAd5 vector-based heterologous prime-boost COVID-19 vaccine in two formulations: two open, non-randomised phase 1/2 studies from Russia. Lancet (London, England) 2020;396(10255):887-97. doi: 10.1016/s0140-6736(20)31866-3 [published Online First: 2020/09/09]
3. Logunov DY, Dolzhikova IV, Shcheblyakov DV, et al. Safety and efficacy of an rAd26 and rAd5 vector-based heterologous prime-boost COVID-19 vaccine: an interim analysis of a randomised controlled phase 3 trial in Russia. Lancet (London, England) 2021;397(10275):671-81. doi: 10.1016/s0140-6736(21)00234-8 [published Online First: 2021/02/06]
4. Bucci E, Andreev K, Björkman A, et al. Safety and efficacy of the Russian COVID-19 vaccine: more information needed. Lancet (London, England) 2020;396(10256):e53. doi: 10.1016/s0140-6736(20)31960-7 [published Online First: 2020/09/25]
5. Logunov DY, Dolzhikova IV, Tukhvatullin AI, et al. Safety and efficacy of the Russian COVID-19 vaccine: more information needed - Authors' reply. Lancet (London, England) 2020;396(10256):e54-e55. doi: 10.1016/s0140-6736(20)31970-x [published Online First: 2020/09/25]
6. The first interim data analysis of the Sputnik V vaccine against COVID-19 phase III clinical trials in the Russian Federation demonstrated 92% efficacy. 2020 [Available from: https://sputnikvaccine.com/newsroom/pressreleases/the-first-interim-data... accessed 03/04/2021 2021.
7. Second interim analysis of clinical trial data showed a 91.4% efficacy for the Sputnik V vaccine on day 28 after the first dose; vaccine efficacy is over 95% 42 days after the first dose. 2020 [Available from: https://sputnikvaccine.com/newsroom/pressreleases/second-interim-analysi... accessed 03/04/2021 2021.
8. The Sputnik V vaccine's efficacy is confirmed at 91.4% based on data analysis of the final control point of clinical trials. 2020 [Available from: https://sputnikvaccine.com/newsroom/pressreleases/the-sputnik-v-vaccine-....
Competing interests:
All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author): Johannes Berkhof, André Gillibert, Gowri Gopalakrishna, Raffaele A Calogero, Anders Bjorkman, Lex M. Bouter, Konstantin Andreev, Florian Naudet, Vasiliy Vlassov have nothing to disclose. Enrico M. Bucci owns Resis Srl, a small business dedicated to the analysis of the scientific literature for the assessment of research integrity.
Enrico M. Bucci (Sbarro Institute - Temple University Department of Biology, USA), Johannes Berkhof (Amsterdam University Medical Centers, Department of Epidemiology and Data Science, Vrije Universiteit Amsterdam, Netherlands), André Gillibert (Department of Biostatistics, CHU Rouen, France), Gowri Gopalakrishna (Amsterdam University Medical Centers, Department of Epidemiology and Data Science, Vrije Universiteit Amsterdam, Netherlands), Raffaele A Calogero (Department of Molecular Biotechnology and Health Sciences, University of Torino, Italy), Anders Bjorkman (Department of Microbiology, Tumor and Cell Biology, Karolinska Institutet, Stockholm, Sweden), Lex M. Bouter (Amsterdam University Medical Centers, Department of Epidemiology and Data Science, Vrije Universiteit Amsterdam, Faculty of Humanities, Department of Philosophy, Netherlands), Konstantin Andreev (Howard Hughes Medical Institute, Department of Molecular Biosciences, Northwestern University, Evanston, USA), Florian Naudet (Univ Rennes, CHU Rennes, Inserm, CIC 1414 (Centre d’Investigation Clinique de Rennes), France), Vasiliy Vlassov (HSE University, Moscow, Russia)
Rapid Response:
Concerns with the Sputnik V vaccine data
Dear Editor
In a commissioned commentary, Chris Baraniuk reviews the “knowns and unknowns” about Russian vaccines against Covid-19, with a specific focus on Sputnik V [1]. While the commentary correctly emphasizes the inconsistencies identified in the phase 1/2 trial results published in the Lancet [2], it mainly discusses the more recently published phase 3 trial results [3].
Our previous concerns regarding the phase 1/2 trial included problematic data patterns with an excess homogeneity of vaccine efficacy across different time points [4]. The authors responded that the unusual data pattern was “a coincidence” due to the small sample size of their study and the discrete distributions of their outcomes [5].
Following such a reasoning, inconsistencies should not be expected in the subsequent larger phase 3 trial. However, we noticed an unexpected homogeneity of vaccine efficacy, this time between age groups. This analysis is central in the Lancet paper at issue because of the disproportionate disease burden in older people. Of course, implausible results can still be observed by chance. However, we have also identified a similar feature, i.e. an excessive homogeneity of the reported vaccine efficacy in the values reported in earlier interim analyses and the published article.
On 11 November 2020, a first press release announced a 92 % efficacy [6]. From this press release we can compute that there were four Covid cases in the vaccine group and 16 in the placebo group. On 24 November 2020, a second press release announced a 91% efficacy with 8/14,095 cases in the vaccine group and 31/4,699 in the placebo group [7]. On 14 December 2020 a third press release announced again a 91% efficacy with 16/17,032 cases in the vaccine group and 62/5,682 in the placebo group [8]. Much to our surprise, the number of cases was exactly twice the number of cases observed in the previous press release [7] in both the placebo and the vaccine groups and about fourfold the number of cases in the vaccine and placebo groups in the first press release [6]. Lastly, the Lancet publication, with a database lock on 24/11/2020, announced a 92% efficacy with 16/14,964 cases in the vaccine group and 62/4,902 in the placebo group [3]. Altogether these results again appear “too good to be true”. The ratio of the number of events between the placebo and vaccine groups is as homogeneous as can be permitted by the number of events (20, 39 and 78) in the press releases [6-8].
The bootstrap P-value for assessing homogeneity of vaccine efficacy between interim stages by means of Pearson’s goodness-of-fit statistic was .9864, taking into account that the total number of events per interim stage is fixed. The Breslow-Day P-value for assessing the homogeneity of vaccine efficacy in age subgroups is equal to .9963. This means that vaccine efficacies are excessively similar between interim analyses and age subgroups even if we were to assume perfect homogeneity. The unusual and improbable high homogeneity of the vaccine efficacy across age strata and different interim analyses raises concerns about the data reported.
In a recent letter to The Lancet documenting these concerns (this letter was submitted upon invitation by The Lancet), the editorial team agreed to publish the letter on condition that we removed our concerns on the press release results since these results were not part of the Lancet publication. We, however felt it important to forward our concerns to the European Medicine Agency which we did on 03/12/2021. The recent BMJ publication by Baraniuk C [1] highlights the phase 3 data but did not address these concerns. We therefore felt it pertinent to inform readers of also these concerns with the Sputnik V vaccine data.
Enrico M. Bucci (Sbarro Institute - Temple University Department of Biology, USA), Johannes Berkhof (Amsterdam University Medical Centers, Department of Epidemiology and Data Science, Vrije Universiteit Amsterdam, Netherlands), André Gillibert (Department of Biostatistics, CHU Rouen, France), Gowri Gopalakrishna (Amsterdam University Medical Centers, Department of Epidemiology and Data Science, Vrije Universiteit Amsterdam, Netherlands), Raffaele A Calogero (Department of Molecular Biotechnology and Health Sciences, University of Torino, Italy), Anders Bjorkman (Department of Microbiology, Tumor and Cell Biology, Karolinska Institutet, Stockholm, Sweden), Lex M. Bouter (Amsterdam University Medical Centers, Department of Epidemiology and Data Science, Vrije Universiteit Amsterdam, Faculty of Humanities, Department of Philosophy, Netherlands), Konstantin Andreev (Howard Hughes Medical Institute, Department of Molecular Biosciences, Northwestern University, Evanston, USA), Florian Naudet (Univ Rennes, CHU Rennes, Inserm, CIC 1414 (Centre d’Investigation Clinique de Rennes), France), Vasiliy Vlassov (HSE University, Moscow, Russia)
REFERENCES
1. Baraniuk C. Covid-19: What do we know about Sputnik V and other Russian vaccines? BMJ (Clinical research ed) 2021;372:n743. doi: 10.1136/bmj.n743 [published Online First: 2021/03/21]
2. Logunov DY, Dolzhikova IV, Zubkova OV, et al. Safety and immunogenicity of an rAd26 and rAd5 vector-based heterologous prime-boost COVID-19 vaccine in two formulations: two open, non-randomised phase 1/2 studies from Russia. Lancet (London, England) 2020;396(10255):887-97. doi: 10.1016/s0140-6736(20)31866-3 [published Online First: 2020/09/09]
3. Logunov DY, Dolzhikova IV, Shcheblyakov DV, et al. Safety and efficacy of an rAd26 and rAd5 vector-based heterologous prime-boost COVID-19 vaccine: an interim analysis of a randomised controlled phase 3 trial in Russia. Lancet (London, England) 2021;397(10275):671-81. doi: 10.1016/s0140-6736(21)00234-8 [published Online First: 2021/02/06]
4. Bucci E, Andreev K, Björkman A, et al. Safety and efficacy of the Russian COVID-19 vaccine: more information needed. Lancet (London, England) 2020;396(10256):e53. doi: 10.1016/s0140-6736(20)31960-7 [published Online First: 2020/09/25]
5. Logunov DY, Dolzhikova IV, Tukhvatullin AI, et al. Safety and efficacy of the Russian COVID-19 vaccine: more information needed - Authors' reply. Lancet (London, England) 2020;396(10256):e54-e55. doi: 10.1016/s0140-6736(20)31970-x [published Online First: 2020/09/25]
6. The first interim data analysis of the Sputnik V vaccine against COVID-19 phase III clinical trials in the Russian Federation demonstrated 92% efficacy. 2020 [Available from: https://sputnikvaccine.com/newsroom/pressreleases/the-first-interim-data... accessed 03/04/2021 2021.
7. Second interim analysis of clinical trial data showed a 91.4% efficacy for the Sputnik V vaccine on day 28 after the first dose; vaccine efficacy is over 95% 42 days after the first dose. 2020 [Available from: https://sputnikvaccine.com/newsroom/pressreleases/second-interim-analysi... accessed 03/04/2021 2021.
8. The Sputnik V vaccine's efficacy is confirmed at 91.4% based on data analysis of the final control point of clinical trials. 2020 [Available from: https://sputnikvaccine.com/newsroom/pressreleases/the-sputnik-v-vaccine-....
Competing interests: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author): Johannes Berkhof, André Gillibert, Gowri Gopalakrishna, Raffaele A Calogero, Anders Bjorkman, Lex M. Bouter, Konstantin Andreev, Florian Naudet, Vasiliy Vlassov have nothing to disclose. Enrico M. Bucci owns Resis Srl, a small business dedicated to the analysis of the scientific literature for the assessment of research integrity.