The Importance of Testing for Blindness in COVID-19 Vaccination Trials
Dear Editor,
Double-blind randomized controlled trials (RCTs) became mandatory for FDA approval of new medications around 1980. This is still a discipline in evolution. As it turns out, blindness breach is rife, although seldom consciously apparent. Shapiro et al. found in a meta-analysis that at the end of double-blind RCTs, treatment assumptions (TAs) were correct in 72-100% of cases [1]. TAs are subjects’ intuitive guesses of group allocation to the verum or placebo group. Nevertheless, reporting upon the success of blinding is uncommon. We didn’t find any in vaccination trials.
Until now, two mRNA COVID-19 vaccination trials have provided sufficient results for FDA approval. Pfizer developed a vaccine with BioNTech and found among 44.000 volunteers in the phase 3 study, 170 confirmed infections. A press-release reported 95% efficacy (94% in people over 65), with 162 cases in the placebo group, 8 in the vaccine group. Out of 10 severe cases, 9 had received a placebo. In the Moderna trial, 95 infections were confirmed; 5 among 15.000 vaccinated subjects got slightly ill, while 79 got slightly and 11 got severely ill among the 15.000 in the placebo arm. In both trials, a diagnosis of infection was made based on participants’ reporting symptoms typical of COVID-19. Symptomatic subjects were molecularly confirmed for SARS-CoV-2.
Placebo related issues
The two studies compare a verum to a passive placebo (injecting 0.9% sodium chloride solution). Compared to an active placebo, a passive placebo can have fewer adverse events. In the case of RNA vaccines, using a passive placebo causes a risk of a breach in blinding, given the consciously and non-consciously felt adverse events of the verum, potentially heightening the “placebo effect” in the verum study arm. This may be of particular concern here because measuring efficacy is based on patient-reported symptoms. In both studies, nonblinded administrators further risk diminishing the double-blind through non-conscious communication.
Being volunteers, subjects may have participated because they are believers in the vaccines’ efficacy, thus having higher prior expectations. In another study where such were assessed, expectancy and desire for pain relief accounted for +/- 80% of the variance in visceral pain intensity [2]. Volunteer vaccine believers might be more attentive to adverse events, increasing the risk of unblinding. Also, they may feel fewer symptoms from COVID-19 if they assume they got the vaccine, potentially leading to a higher placebo effect and reporting bias. Prior expectations are also enhanced through administration as an injection. Many studies show that the more ‘special’ the administration, the higher the placebo effect.
Why assess treatment assumption?
Taking TAs into account has not reached routine practice. Measuring TA in COVID-19 vaccine trials would provide insight into study blindness. Because case confirmation by molecular testing happens only after the appearance of subjective symptoms, a blindness breach could potentially have an influence on overall efficacy through the placebo effect, attentional factors, and reporting bias. Also, a nocebo effect from the idea of not being in the verum group (= low TA) PLUS being symptomatic PLUS receiving a molecular diagnosis in the early symptomatic stage could make mild symptoms more severe.
TA data can be used to improve the scientific validity, and indicate whether an active placebo is needed. Assessing TAs is interesting for vaccine producers. They are relatively low-cost additions that make trials more efficient.
If the general influence of the mind on COVID-19 symptoms is substantial – to which a strong placebo effect would be one indication – investigations into using this may be worthwhile, as has been argued for COVID-19 elsewhere. We have witnessed several premature claims of drugs against COVID-19, which disappeared upon large double-blinded clinical trials [3].
Using TA assessment during vaccination trials may lead to better insights into mind influences not only on COVID-19 progression but also on a vaccine’s effect, given the mind’s impact on inflammation and the immune system [4]. Can positive mental factors boost the vaccination effect? Can the mind-immuno-viral interplay affect the seriousness of COVID-19? People being complex systems, one expects non-linear phenomena; the mind can be more impactful than expected in linear thinking. Assessing TA in trials may pave the way to some answers.
Given that we don’t know enough about the duration of protection, long-term adverse events or the risk of vaccine-related hypersensitivity, any better assessment of effectiveness can have an impact on vaccination strategies.
When and how to perform TA assessment
TAs can be easily assessed at different stages during the trial. The way of asking is essential: concise and to the point to better reveal also non-conscious assumptions. We propose to do this right after vaccine injections; two days after each injection with adverse events fresh in memory; concomitant with other evaluations in order to cut costs; at the appearance of symptoms. Additionally, we propose to assess prior expectations before the first administration.
Conclusions
The present vaccines’ reported efficacy relates to the prevention of COVID-19 symptoms, while the value for society is efficacy to prevent infection, reduce infectiousness, or prevent serious COVID-19 cases.
The reported high efficacy may not translate into similarly high effectiveness for various reasons that have been discussed. The impact of breaching blindness has not received proper attention. Passive placebo may facilitate a breach in blindness. Thus, the placebo effect, together with reporting bias, may have influenced the reported efficacy. Testing TA gives more accurate assessments of effectiveness by uncovering potential bias5.
Assessing TA can additionally help to understand to what extent the mind is important in virus or vaccine immune response and COVID-19 progression. If this is substantial, it can contribute to treatment. There may be a positive effect on the patient and on the effectiveness of the vaccine itself.
Incorporating TA in a study is a relatively low-cost procedure with a high possible yield. It can be included in future COVID-19 vaccination trials from the start and in ongoing studies as soon as possible.
Dr. Jean-Luc Mommaerts, M.D., M.A.I., - Ph.D. Jean-Luc.Mommaerts@vub.be - Vrije Universiteit Brussel (Free University Brussels)
Prof. dr. Dirk Devroey, M.D., Ph.D., Dean of Medicine & Farmacy - dirk.devroey@vub.be - Vrije Universiteit Brussel (Free University Brussels)
Prof. Dr. Anne-Mieke Vandamme, Ph.D. - annemie.vandamme@kuleuven.be - KU Leuven - Clinical and Epidemiological Virology - Rega Institute for Medical Research
References
(1) Shapiro AK, Shapiro E. The Powerful Placebo. Baltimore: The Johns Hopkins University Press, 1997.
(2) Vase L, Robinson ME, Verne GN, Price DD. The contributions of suggestion, desire, and expectation to placebo effects in irritable bowel syndrome patients. An empirical investigation. Pain. 2003 Sep;105(1-2):17-25. doi: 10.1016/s0304-3959(03)00073-3. PMID: 14499416.
(3) Hsu J. Covid-19: What now for remdesivir? BMJ 2020;371:m4457.
(4) Brod S, Rattazzi L, Piras G, D’Acquisto F. ‘As above, so below’ examining the interplay between emotion and the immune system. Immunology. 2014;143(3):311-318. doi:10.1111/imm.12341
(5) Colagiuri B, Benedetti F. Testing for blinding is the only way to determine whether a trial is blind. Rapid response to: CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c332.
Competing interests: Anne-Mieke Vandamme has received consultancy fees from Gilead for their virology advisory board.
Yours sincerely,
Jean-Luc Mommaerts
Competing interests:
Anne-Mieke Vandamme has received consultancy fees from Gilead for their virology advisory board.
Rapid Response:
The Importance of Testing for Blindness in COVID-19 Vaccination Trials
Dear Editor,
Double-blind randomized controlled trials (RCTs) became mandatory for FDA approval of new medications around 1980. This is still a discipline in evolution. As it turns out, blindness breach is rife, although seldom consciously apparent. Shapiro et al. found in a meta-analysis that at the end of double-blind RCTs, treatment assumptions (TAs) were correct in 72-100% of cases [1]. TAs are subjects’ intuitive guesses of group allocation to the verum or placebo group. Nevertheless, reporting upon the success of blinding is uncommon. We didn’t find any in vaccination trials.
Until now, two mRNA COVID-19 vaccination trials have provided sufficient results for FDA approval. Pfizer developed a vaccine with BioNTech and found among 44.000 volunteers in the phase 3 study, 170 confirmed infections. A press-release reported 95% efficacy (94% in people over 65), with 162 cases in the placebo group, 8 in the vaccine group. Out of 10 severe cases, 9 had received a placebo. In the Moderna trial, 95 infections were confirmed; 5 among 15.000 vaccinated subjects got slightly ill, while 79 got slightly and 11 got severely ill among the 15.000 in the placebo arm. In both trials, a diagnosis of infection was made based on participants’ reporting symptoms typical of COVID-19. Symptomatic subjects were molecularly confirmed for SARS-CoV-2.
Placebo related issues
The two studies compare a verum to a passive placebo (injecting 0.9% sodium chloride solution). Compared to an active placebo, a passive placebo can have fewer adverse events. In the case of RNA vaccines, using a passive placebo causes a risk of a breach in blinding, given the consciously and non-consciously felt adverse events of the verum, potentially heightening the “placebo effect” in the verum study arm. This may be of particular concern here because measuring efficacy is based on patient-reported symptoms. In both studies, nonblinded administrators further risk diminishing the double-blind through non-conscious communication.
Being volunteers, subjects may have participated because they are believers in the vaccines’ efficacy, thus having higher prior expectations. In another study where such were assessed, expectancy and desire for pain relief accounted for +/- 80% of the variance in visceral pain intensity [2]. Volunteer vaccine believers might be more attentive to adverse events, increasing the risk of unblinding. Also, they may feel fewer symptoms from COVID-19 if they assume they got the vaccine, potentially leading to a higher placebo effect and reporting bias. Prior expectations are also enhanced through administration as an injection. Many studies show that the more ‘special’ the administration, the higher the placebo effect.
Why assess treatment assumption?
Taking TAs into account has not reached routine practice. Measuring TA in COVID-19 vaccine trials would provide insight into study blindness. Because case confirmation by molecular testing happens only after the appearance of subjective symptoms, a blindness breach could potentially have an influence on overall efficacy through the placebo effect, attentional factors, and reporting bias. Also, a nocebo effect from the idea of not being in the verum group (= low TA) PLUS being symptomatic PLUS receiving a molecular diagnosis in the early symptomatic stage could make mild symptoms more severe.
TA data can be used to improve the scientific validity, and indicate whether an active placebo is needed. Assessing TAs is interesting for vaccine producers. They are relatively low-cost additions that make trials more efficient.
If the general influence of the mind on COVID-19 symptoms is substantial – to which a strong placebo effect would be one indication – investigations into using this may be worthwhile, as has been argued for COVID-19 elsewhere. We have witnessed several premature claims of drugs against COVID-19, which disappeared upon large double-blinded clinical trials [3].
Using TA assessment during vaccination trials may lead to better insights into mind influences not only on COVID-19 progression but also on a vaccine’s effect, given the mind’s impact on inflammation and the immune system [4]. Can positive mental factors boost the vaccination effect? Can the mind-immuno-viral interplay affect the seriousness of COVID-19? People being complex systems, one expects non-linear phenomena; the mind can be more impactful than expected in linear thinking. Assessing TA in trials may pave the way to some answers.
Given that we don’t know enough about the duration of protection, long-term adverse events or the risk of vaccine-related hypersensitivity, any better assessment of effectiveness can have an impact on vaccination strategies.
When and how to perform TA assessment
TAs can be easily assessed at different stages during the trial. The way of asking is essential: concise and to the point to better reveal also non-conscious assumptions. We propose to do this right after vaccine injections; two days after each injection with adverse events fresh in memory; concomitant with other evaluations in order to cut costs; at the appearance of symptoms. Additionally, we propose to assess prior expectations before the first administration.
Conclusions
The present vaccines’ reported efficacy relates to the prevention of COVID-19 symptoms, while the value for society is efficacy to prevent infection, reduce infectiousness, or prevent serious COVID-19 cases.
The reported high efficacy may not translate into similarly high effectiveness for various reasons that have been discussed. The impact of breaching blindness has not received proper attention. Passive placebo may facilitate a breach in blindness. Thus, the placebo effect, together with reporting bias, may have influenced the reported efficacy. Testing TA gives more accurate assessments of effectiveness by uncovering potential bias5.
Assessing TA can additionally help to understand to what extent the mind is important in virus or vaccine immune response and COVID-19 progression. If this is substantial, it can contribute to treatment. There may be a positive effect on the patient and on the effectiveness of the vaccine itself.
Incorporating TA in a study is a relatively low-cost procedure with a high possible yield. It can be included in future COVID-19 vaccination trials from the start and in ongoing studies as soon as possible.
Dr. Jean-Luc Mommaerts, M.D., M.A.I., - Ph.D. Jean-Luc.Mommaerts@vub.be - Vrije Universiteit Brussel (Free University Brussels)
Prof. dr. Dirk Devroey, M.D., Ph.D., Dean of Medicine & Farmacy - dirk.devroey@vub.be - Vrije Universiteit Brussel (Free University Brussels)
Prof. Dr. Anne-Mieke Vandamme, Ph.D. - annemie.vandamme@kuleuven.be - KU Leuven - Clinical and Epidemiological Virology - Rega Institute for Medical Research
References
(1) Shapiro AK, Shapiro E. The Powerful Placebo. Baltimore: The Johns Hopkins University Press, 1997.
(2) Vase L, Robinson ME, Verne GN, Price DD. The contributions of suggestion, desire, and expectation to placebo effects in irritable bowel syndrome patients. An empirical investigation. Pain. 2003 Sep;105(1-2):17-25. doi: 10.1016/s0304-3959(03)00073-3. PMID: 14499416.
(3) Hsu J. Covid-19: What now for remdesivir? BMJ 2020;371:m4457.
(4) Brod S, Rattazzi L, Piras G, D’Acquisto F. ‘As above, so below’ examining the interplay between emotion and the immune system. Immunology. 2014;143(3):311-318. doi:10.1111/imm.12341
(5) Colagiuri B, Benedetti F. Testing for blinding is the only way to determine whether a trial is blind. Rapid response to: CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c332.
Competing interests: Anne-Mieke Vandamme has received consultancy fees from Gilead for their virology advisory board.
Yours sincerely,
Jean-Luc Mommaerts
Competing interests: Anne-Mieke Vandamme has received consultancy fees from Gilead for their virology advisory board.