Modelling the pandemic

Dear Editor,

In this response, I take issue with the pessimistic view of Devi Sridhar and Maimuna Majumder [1] on axiomatic modeling in epidemiology and defend this way of informing nonpharmaceutical interventions. In "Modelling the pandemic," the authors claim that our current mitigation and suppression measures deployed to cope with the SARS-Cov-2 outbreak are guided by (and over-rely on) mathematical models. They highlight the reliance of mathematical models on certain assumptions and point out at the value-ladenness of decisions regarding the inclusion or exclusion of factors from such models. As a solution, the authors advise making both data and code publicly available to ensure replicability and supporting modeling with other types of evidence such as case studies of policies implemented in other countries, anecdotal evidence from frontline healthcare workers, and studies of previous epidemics.

Even though I lean towards the view that supporting decisions with different types of evidence is beneficial, I am concerned that we currently lack studies that assess the efficacy of nonpharmaceutical interventions other than the above mentioned models. The first study [1] employing natural experiment design to assess the response in China was published weeks after similar measures had to be introduced in other countries. Using evidence from previous outbreaks, while possibly informative, requires extrapolating results onto the current epidemic because each pathogen is characterized by specific features and infectiousness, leading to pandemics having different and sometimes unique characteristics. The data needed to identify a degree of similarity between SARS-CoV-2 and some previous outbreak is likely to be sufficient for establishing efficacy claims on its own (known in philosophical literature as the extrapolator's circle). Moreover, even though the first-hand experiences of healthcare professionals may be very informative concerning caring for Covid-19 patients (e.g., such as suggesting prone positioning [3]), they are unreliable in assessing nonpharmaceutical interventions targeting social behavior and those which are not in a healthcare environment.

Since this is the situation we find ourselves in, the mitigation responses have to be informed by axiomatic models. Moreover, the lack of empirical evidence is likely to constrain the quality of decisions regarding mitigation measures. Regrettably, even though epidemiologists have used compartmental models (such as SIR – susceptible, infectious, and recovered) for over a century, they have never been considered in the context of evidence-based medicine. None of the standard evidence hierarchies and assessment tools explicitly mention axiomatic models. The Oxford Centre for Evidence-Based Medicine [4] lists mechanism-based reasoning at the lowest stage, and NICE guidelines [5] do not include epidemiological models or mechanistic reasoning. This indicates the need to consider what type of evidence (if any) can be delivered by axiomatic models. Is this type of evidence as reliable as RCTs, or does its exclusion from the hierarchies of evidence stem from a lack of faith in their functioning?

A clarification is in order here. Sridhar and Majumder, despite exemplifying their discussion with an agent-based model, focus on mathematical models. While the SIR framework and its more advanced derivatives (which are the primary group of mathematical epidemiological models) have been used for predicting the development of the epidemic [6] [7], policy decisions have actually been led [8] by agent-based models (ABMs) [9] [10]. They differ from mathematical models in terms of both their advancement and use. First, ABMs cannot be solved mathematically and require the use of computational methods (e.g., simulation). Second, the most advanced ABMs currently simulate the society of agents exposed to the risk of infection during social interactions in different contexts (at household, work, community, etc.). The input to ABMs differs significantly from the axioms of standard compartmental models. The rules setting the behavior of agents are calibrated to statistical data describing work and commuting habits, travel distances, and neighborhood interactions, making the society of agents similar (in epidemiologically relevant aspects) to the societies represented with these models. The assumptions describing the infectiousness of the pathogen are also informed empirically. Often, calibration to the epidemiological data from the beginning of the outbreak is applied.

It remains true for ABMs that "[a]ll models are limited by the assumptions that they make," however, this alone is insufficient grounds for skepticism regarding this method, especially if no other evidence is present and decisions need to be made. All results in medicine rely on assumptions. In vitro and animal studies rely on the assumption that the process producing results also operates in humans. Meta-analyses assume the homogeneity of randomized trials under investigation. Similarly, the accusation that “mathematical models do not include value systems or morals” describes all research methods used in medicine or elsewhere. RCTs can inform us of the efficacy or safety of a drug in comparison to a control group, but they are neither suitable nor aimed at weighing the risk-reward ratio. Observational studies in epidemiology can deliver evidence that exposure is related to a negative outcome, but only nonepistemic values external to research can inform the decision if the potential benefit outweighs the (also non-financial) costs of limiting exposure.

How, therefore, should we assess the quality of evidence for the efficacy of mitigation and suppression measures delivered by epidemiological ABMs? Such models, considered as representing existing mechanisms [11], can be compared to laboratory research. The efficacy claims are internally valid as long as no coding or calculation error occurs (i.e., they accurately describe the effects of interventions within the model world). However, if the mechanisms represented by ABMs differ significantly from the process producing the outbreak that is targeted with the intervention under consideration or is influenced by factors external to the model, then such evidence, similarly to a significant number of laboratory studies, lack external validity. Unfortunately, considering the fact that RCTs are not feasible for assessing the efficacy of nonpharmaceutical interventions and the decisions need to be made before observational results will be accessible, decision-makers should consider the current best evidence and assess the external validity of epidemiological models in order to choose the study most relevant for their purpose.

References:
[1] Sridhar, D, Majumder, D. Modelling the pandemic. BMJ. 2020: 369:m1567. doi: https://doi.org/10.1136/bmj.m1567
[2] Pan A, Liu L, Wang C, Guo H, Hao X, Wang Q, Huang J, He N, Yu H, Lin X, Wei S. Association of public health interventions with the epidemiology of the COVID-19 outbreak in Wuhan, China. JAMA. 2020 Apr 10.
[3] Ghelichkhani P, Esmaeili M. Prone Position in Management of COVID-19 Patients; a Commentary. Archives of Academic Emergency Medicine. 2020;8(1).
[4] OCEBM Levels of Evidence Working Group. " The Oxford 2011 Levels of Evidence." Oxford Centre for Evidence-Based Medicine. http://www. cebm. net/index. aspx? o= 5653. 2011.
[5] National Institute for Health and Care Excellence. Developing NICE guidelines: the manual. https://www.nice.org.uk/media/default/about/what-we-do/our-programmes/de...
[6] Koo JR, Cook AR, Park M, Sun Y, Sun H, Lim JT, Tam C, Dickens BL. Interventions to mitigate early spread of SARS-CoV-2 in Singapore: a modelling study. The Lancet Infectious Diseases. 2020 Mar 23. https://www.thelancet.com/journals/lancet/article/PIIS1473-3099(20)30162-6/fulltext
[7] Giordano G, Blanchini F, Bruno R, Colaneri P, Di Filippo A, Di Matteo A, Colaneri M. A SIDARTHE model of COVID-19 epidemic in Italy. arXiv preprint arXiv:2003.09861. 2020 Mar 22.
[8] Adam D. Special report: The simulations driving the world's response to COVID-19. Nature. 2020 Apr 2. https://www.nature.com/articles/d41586-020-01003-6
[9] Ferguson N, Laydon D, Nedjati Gilani G, Imai N, Ainslie K, Baguelin M, Bhatia S, Boonyasiri A, Cucunuba Perez ZU, Cuomo-Dannenburg G, Dighe A. Report 9: Impact of non-pharmaceutical interventions (NPIs) to reduce COVID19 mortality and healthcare demand. Imperial College London.
[10] Chang SL, Harding N, Zachreson C, Cliff OM, Prokopenko M. Modelling transmission and control of the COVID-19 pandemic in Australia. arXiv preprint arXiv:2003.10218. 2020 Mar 23.
[11] Clarke B, Gillies D, Illari P, Russo F, Williamson J. Mechanisms and the evidence hierarchy. Topoi. 2014 Oct 1;33(2):339-60.

The work of Mariusz Maziarz has received funding from the European Research Council (ERC) (grant agreement No 805498).