Masks, media, fact checkers and the interpretation of scientific evidence.
Dear Editor
We have been interested in the interpretation of the statistics related to the recently published trial of the efficacy of masks by Danish academics [1]. The study which tested the efficacy of face masks in addition to usual hygiene measures was by conventional statistical interpretation ‘no effect’ is now the subject of considerable controversy. An editorial by Abbasi for the BMJ has also disagreed with a conventional interpretation of the trial’s results, yet criticised the labelling of the “no effect” reading by Oxford professors Heneghan and Jefferson as “false information” [2].
We drew attention to the results of this study in a post on our website [3] and facebook. We have also been the subject of “fact checkers”, this time from the AAP [4], who have labelled our “no effect” interpretation as having the potential to “mislead without additional context.”
The arguments of experts interviewed by the fact checkers were interesting. For example, Professor Michael Baker, a proponent of mass masking in New Zealand, stated “There is nothing at all surprising about the findings of the Danish study and it no way (sic) alters or contradicts the current understanding of how masks operate for mass masking…” He said the study found masks had a small protective effect, although this was not statistically significant due to the low sample size. Further aligned comments were given by Associate-Professor Siouxsie Wiles: “it was misleading to claim the Danish study found masks were ineffective”. The article went on to claim that observational studies supported the use of masks. [5] We concur that uncertainty about the interpretation of the trial exists and that a variety of understandings of the results are possible, under a Bayesian paradigm in which new evidence updates or alters prior beliefs. However, we believe the controversy over the results of the mask trial raises some interesting points about the interpretation of scientific evidence in the covid-19 era.
First, the study was designed using a frequentist paradigm in which the minimal detectable effect was stated as a halving of infection proportions in the study’s power calculation. The trial’s authors reported that the study had 80% statistical power to reject the “no effect” belief assuming the 50% effect existed. [1] A retrospective power calculation shows that in the event, about 90% power for such an effect was attained.
Thus, a conventional interpretation of the results (odds ratio 0.82 [95% confidence interval: 0.54 to 1.23]) would be that there is insufficient evidence to reject the idea that masks have no influence on preventing the spread of covid-19. The authors themselves have claimed the result may be a type-2 error and lesser degrees of protection are possible. As indicated by the power calculation however, this reading of the evidence is less likely than the conventional one. The P-value from the study (0.3) indicates that the trial’s observed results or more extreme would occur ~1/3 times if the trial were repeated again and again and masks had no effect, indicating that the play of random chance is a likely explanation for the apparent benefit observed. For this reason, P-values of 0.05 are conventionally used to abandon a “no effect” hypothesis.
One could argue that convention should be abandoned for a more optimistic interpretation of trial evidence. However, if we were to follow the same logic we may end up widely promoting ineffective treatments. As an example, electrostimulation, laser therapy and acupuncture are not generally thought to improve smoking cessation success, yet several promising pooled effects were calculated in a meta-analysis, although the majority were not “statistically significant” [6].
The tone of the “fact checking piece” in which the findings of the Danish study apparently support mass masking and that they have a “small protective effect” with a more conventional interpretation labelled as “misleading” turns usual scientific practice on its head, and is outside the a priori rules set up by the researchers themselves in the trial protocol. The pointing to observational evidence to contradict trial results is another subversion of usual epidemiological practice [7]. While this may seem banal and trivial, we see it as an indication of the subtle distortion of results and the politicisation of evidence in the covid-19 era.
References
1. Bundgaard H, Bundgaard JS, Raaschou-Pedersen DET, et al. Effectiveness of adding a mask recommendation to other public health measures to prevent SARS-CoV-2 infection in Danish mask wearers: a randomized controlled trial. Ann Intern Med 2020. doi: 10.7326/M20-68
2. Abbasi K. The curious case of the Danish mask study. BMJ 2020;371:m4586. doi: 10.1136/bmj.m4586
3. Covid Plan B. Danish mask study result; no statistical difference from not wearing one. Wellington: Covid Plan B; 2020 [Available from: https://www.covidplanb.co.nz/our-posts/danish-mask-study-result-no-stati... accessed 1 December 2020.
4. Driver G. Does a new study show masks are ineffective at stopping COVID-19 infection? : AAP; 2020 [Available from: https://www.aap.com.au/does-a-new-study-show-masks-are-ineffective-at-st... accessed 1/12/2020 2020.
5. Center of Disease Control and Prevention. Scientific Brief: Community Use of Cloth Masks to Control the Spread of SARS-CoV-2. Atlanta: CDC.; 2020 [Available from: https://www.cdc.gov/coronavirus/2019-ncov/more/masking-science-sars-cov2... accessed 1 December 2020.
6. White AR, Rampes H, Liu JP, et al. Acupuncture and related interventions for smoking cessation. Cochrane Database of Systematic Reviews 2014(1) doi: 10.1002/14651858.CD000009.pub4
7. Brighton B, Bhandari M, Tornetta P, et al. Hierarchy of evidence: from case reports to randomized controlled trials. Clin Orthop Relat Res 2003;413:19-24.
Competing interests:
No competing interests
02 December 2020
Simon J. Thornley
Senior Lecturer
Michael D. Jackson, Gerhard Sundborn.
University of Auckland
The University of Auckland, Building 507, Park Rd, Grafton, Auckland, New Zealand.
Rapid Response:
Masks, media, fact checkers and the interpretation of scientific evidence.
Dear Editor
We have been interested in the interpretation of the statistics related to the recently published trial of the efficacy of masks by Danish academics [1]. The study which tested the efficacy of face masks in addition to usual hygiene measures was by conventional statistical interpretation ‘no effect’ is now the subject of considerable controversy. An editorial by Abbasi for the BMJ has also disagreed with a conventional interpretation of the trial’s results, yet criticised the labelling of the “no effect” reading by Oxford professors Heneghan and Jefferson as “false information” [2].
We drew attention to the results of this study in a post on our website [3] and facebook. We have also been the subject of “fact checkers”, this time from the AAP [4], who have labelled our “no effect” interpretation as having the potential to “mislead without additional context.”
The arguments of experts interviewed by the fact checkers were interesting. For example, Professor Michael Baker, a proponent of mass masking in New Zealand, stated “There is nothing at all surprising about the findings of the Danish study and it no way (sic) alters or contradicts the current understanding of how masks operate for mass masking…” He said the study found masks had a small protective effect, although this was not statistically significant due to the low sample size. Further aligned comments were given by Associate-Professor Siouxsie Wiles: “it was misleading to claim the Danish study found masks were ineffective”. The article went on to claim that observational studies supported the use of masks. [5] We concur that uncertainty about the interpretation of the trial exists and that a variety of understandings of the results are possible, under a Bayesian paradigm in which new evidence updates or alters prior beliefs. However, we believe the controversy over the results of the mask trial raises some interesting points about the interpretation of scientific evidence in the covid-19 era.
First, the study was designed using a frequentist paradigm in which the minimal detectable effect was stated as a halving of infection proportions in the study’s power calculation. The trial’s authors reported that the study had 80% statistical power to reject the “no effect” belief assuming the 50% effect existed. [1] A retrospective power calculation shows that in the event, about 90% power for such an effect was attained.
Thus, a conventional interpretation of the results (odds ratio 0.82 [95% confidence interval: 0.54 to 1.23]) would be that there is insufficient evidence to reject the idea that masks have no influence on preventing the spread of covid-19. The authors themselves have claimed the result may be a type-2 error and lesser degrees of protection are possible. As indicated by the power calculation however, this reading of the evidence is less likely than the conventional one. The P-value from the study (0.3) indicates that the trial’s observed results or more extreme would occur ~1/3 times if the trial were repeated again and again and masks had no effect, indicating that the play of random chance is a likely explanation for the apparent benefit observed. For this reason, P-values of 0.05 are conventionally used to abandon a “no effect” hypothesis.
One could argue that convention should be abandoned for a more optimistic interpretation of trial evidence. However, if we were to follow the same logic we may end up widely promoting ineffective treatments. As an example, electrostimulation, laser therapy and acupuncture are not generally thought to improve smoking cessation success, yet several promising pooled effects were calculated in a meta-analysis, although the majority were not “statistically significant” [6].
The tone of the “fact checking piece” in which the findings of the Danish study apparently support mass masking and that they have a “small protective effect” with a more conventional interpretation labelled as “misleading” turns usual scientific practice on its head, and is outside the a priori rules set up by the researchers themselves in the trial protocol. The pointing to observational evidence to contradict trial results is another subversion of usual epidemiological practice [7]. While this may seem banal and trivial, we see it as an indication of the subtle distortion of results and the politicisation of evidence in the covid-19 era.
References
1. Bundgaard H, Bundgaard JS, Raaschou-Pedersen DET, et al. Effectiveness of adding a mask recommendation to other public health measures to prevent SARS-CoV-2 infection in Danish mask wearers: a randomized controlled trial. Ann Intern Med 2020. doi: 10.7326/M20-68
2. Abbasi K. The curious case of the Danish mask study. BMJ 2020;371:m4586. doi: 10.1136/bmj.m4586
3. Covid Plan B. Danish mask study result; no statistical difference from not wearing one. Wellington: Covid Plan B; 2020 [Available from: https://www.covidplanb.co.nz/our-posts/danish-mask-study-result-no-stati... accessed 1 December 2020.
4. Driver G. Does a new study show masks are ineffective at stopping COVID-19 infection? : AAP; 2020 [Available from: https://www.aap.com.au/does-a-new-study-show-masks-are-ineffective-at-st... accessed 1/12/2020 2020.
5. Center of Disease Control and Prevention. Scientific Brief: Community Use of Cloth Masks to Control the Spread of SARS-CoV-2. Atlanta: CDC.; 2020 [Available from: https://www.cdc.gov/coronavirus/2019-ncov/more/masking-science-sars-cov2... accessed 1 December 2020.
6. White AR, Rampes H, Liu JP, et al. Acupuncture and related interventions for smoking cessation. Cochrane Database of Systematic Reviews 2014(1) doi: 10.1002/14651858.CD000009.pub4
7. Brighton B, Bhandari M, Tornetta P, et al. Hierarchy of evidence: from case reports to randomized controlled trials. Clin Orthop Relat Res 2003;413:19-24.
Competing interests: No competing interests