Role of mathematical modelling in future pandemic response policyBMJ 2022; 378 doi: https://doi.org/10.1136/bmj-2022-070615 (Published 15 September 2022) Cite this as: BMJ 2022;378:e070615
- Christina Pagel, professor of operational research1,
- Christian A Yates, senior lecturer in mathematical biology2
- 1Clinical Operational Research Unit, University College London, London, UK
- 2University of Bath, Bath, UK
- Correspondence to: C Pagel
Mathematical modelling is intrinsically difficult given the complexity of relationships between parameters and difficulty quantifying those parameters
Modelling needs input from a much wider range of sources including domain experts
Data sharing and communication of results could be improved
Policy makers and the public often had poor understanding of key concepts such as exponential growth and the limitations of long-term forecasting
Mathematical modelling underpinned much of the advice that the Scientific Advisory Group for Emergencies (SAGE) and others provided to the UK government during the pandemic. It should therefore be a focus of the UK covid inquiry’s examination of how and why decisions were made.1 Much of the modelling came from the Scientific Pandemic Influenza Group on Modelling (SPI-M), which gives expert advice to the Department of Health and Social Care and the wider UK government on emerging human infectious disease threats. Its members come from a range of UK institutions and their advice is based on infectious disease modelling and epidemiology.2 During the pandemic, SPI-M reported to SAGE.
Modelling from the group has been influential throughout the pandemic, particularly during the first 18 months. For instance, “report 9” by the Imperial College modelling group3 was an important trigger for the UK government’s decision to implement a nationwide lockdown in March 2020. Projections from multiple independent modelling teams also informed the UK’s “roadmap” for release from lockdown in February 20214 and implementation of some further public health mandates under “plan B” measures in December 2021 during the first omicron wave.5 Modelling determined the vaccine priority groups in December 2020,6 which contributed to the UK’s successful vaccine rollout and consequently saved thousands of lives over the first half of 2021. SPI-M’s work has also been important in evaluating the relative effect of different interventions, such as the role of home working in reducing transmission in 2021.7
Throughout the pandemic, official modelling efforts have been criticised from many different quarters. Some of that criticism has been understandable—a result of highly publicised projections that never came to pass (modellers had little control over which projections were amplified in the media). Some missteps derived directly from failures of the modelling process to capture reality—use of inaccurate model parameters because of uncertain data; misunderstanding or misinterpretation of the key features of the situations being modelled; and the intrinsic inability of most models to capture important facets of human behaviour. However, much of the criticism modellers have received has been misplaced, a result of fundamental misunderstandings of the purpose of mathematical modelling, what it is capable of, and how its results should be interpreted. These misunderstandings result, in part, from failures in communication.
This was seen in sustained criticism of SPI-M models on the impact of the omicron variant in the UK in December 2021.89101112 The models turned out to be too pessimistic because of a combination of uncertainty about omicron’s severity and about how the public would react to growing cases. In the end, omicron proved to cause less severe lung disease in adults than delta, the boosters were more effective, and—for the first omicron wave—the public voluntarily restricted their contacts and took up rapid antigen testing much more than expected, which all combined to reduce the wave’s severity.1314 The model assumptions were clear within SPI-M’s reports,151617 but the attacks expanded to cover the whole of SPI-M’s contribution.1819 However, what was not anticipated in the modelling, or in policy, was a second (and just as large) omicron wave just three months after the first. In fact, the combination of two waves in short succession contributed to the worst waits for emergency care 20 since data collection began and high levels of sick leave among staff in the NHS.21
In considering the role of modelling during the pandemic response, the UK covid inquiry needs to consider whether it was based on the right information and how the results were used and communicated. SAGE was not charged with the economic modelling of policy options and that is beyond the remit of this article. The inquiry might like to consider separately whether and how economic modelling could have been part of the SAGE remit.
How mathematical modelling is used to inform policy
Mathematical modelling provides a framework in which we can formalise our assumptions about the processes we are trying to capture (eg, disease spread and impact), build them into a simplified representation of reality, and simulate forward in time to suggest what might happen in the future under different policy options.2223 Modelling is also extremely useful in understanding the underlying situation when we have incomplete or missing data,2425 and can shed light on what has happened in the past when the picture is murky, such as the effect of different public health mitigations.24A detailed review of how SPI-M was formed and how its work feeds into government policy via SAGE was published in 2021.2
Epidemiological modelling is more akin to science than to pure mathematics. The process involves iteratively building models, making predictions, comparing these predictions to observations, and then refining the models. Through this repetitive process modellers can build accurate, detailed, and robust representations of reality, which can then be used to speculate about what will happen in hitherto unseen scenarios. Most applications of mathematical modelling allow for many repeats of this cycle over periods of weeks, months, or even years. By contrast, synthesising appropriate data to populate and fine tune models in real time during a pandemic is an almost unique challenge in applied mathematics.2
Any modelling comes with various uncertainties and assumptions that need to be thought through, examined, and explained.26 Substantial errors in any area can derail the usefulness of the model, and, if not understood and recognised, cause harm. In the context of a rapidly evolving pandemic this is even more important. Good mathematical modelling must be transparent about all the sources of uncertainty (table 1) and provide sufficient detail to outsiders (including policy makers) to assess the model outputs.
Questions for the public inquiry
How can we ensure all the right disciplines and perspectives are included in the modelling efforts?
How can data be generated and shared within and between modelling groups to sustain a more egalitarian and robust modelling environment?
Would better public communication of modelling processes and underlying assumptions improve usefulness, and how can this communication be resourced?
How helpful were pandemic projections looking a year or more ahead?
SPI-M modelling during the pandemic has been admirably transparent about key assumptions and parameter estimates and has typically encompassed a range of scenarios. The models have incorporated inherent variability and highlighted many of the problems associated with unknown future events. Structural details of the SPI-M modelling are usually available in academic papers but are not easily accessible to a non-academic audience. Nonetheless, this transparency has not been sufficient to prevent mistakes or criticism. What then are the key questions around the role of modelling that the public inquiry should address?
Expert input was sometimes too narrow
As described above, model building is iterative. The structure of the model and its input parameters are continuously refined in light of the latest evidence and understanding about the dynamics of the disease and its spread. Perhaps the biggest threat to the usefulness of the models is when important information or knowledge relating to the dynamics is held by experts who are not connected to the modelling community, including the public.27 The modelling related to care homes during the covid-19 pandemic represents perhaps the most important cautionary tale.
Older and sicker populations were known early on to be at much higher risk of severe illness and death from covid-19. Modellers on SPI-M quickly understood that elderly people, and particularly those in care homes, were at high risk should they catch coronavirus. The need for protection of care home residents was also well appreciated, yet surprisingly the words “care homes” appear only twice in SAGE minutes during the first five months of the pandemic.28 Modellers were given access to the excellent hospital surveillance dataset at the start of the pandemic,29 but there was poor understanding of some of the important factors required for models to appropriately represent social care settings and thereby protect care homes. Experts on the care sector identified the intersecting factors of an extremely vulnerable population living in shared accommodation, frequent contact with friends and relatives in the community, the discharge of potentially sick patients from hospitals, the lack of personal protective equipment and low paid staff (with little access to sick pay, working across multiple homes as agency workers and more likely to live in multi-occupancy poor housing) as particular system vulnerabilities. Many of these issues, however, did not seem to be anticipated or explicitly taken into account by the modellers.
Although mathematical modellers could not be expected to have a prior understanding of the details of the social care sector and the interacting features of vulnerable populations and staff, they should have realised that they might be unaware of important factors and needed to seek relevant expertise. To social care experts, the vulnerabilities in the system were both knowable and known. However, modellers seem to have failed to identify the knowledge gap and so could not access relevant knowledge from those with the requisite expertise. Once the vulnerability of care homes became clearer to modellers, their specific features were successfully incorporated into models which then (albeit with hindsight) highlighted the high numbers of deaths if mitigations were not adequate.3031 In England and Wales there were more than 27 000 deaths in care homes during the first wave of the pandemic.32
Modelling subgroups convened by government should draw on as much diverse expertise as possible—an aspect also considered by the Parliamentary Committee on Science and Technology.33 Learning could be drawn from published literature on interdisciplinary working 34 in disaster response353637 and adapted to the UK situation. The mechanisms for ensuring interdisciplinary working must be in place and documented before a pandemic hits and should be agnostic to the nature of the pandemic or to the personal experience and networks of lead experts at the time. This recommendation also applies to the overall structure of SAGE, which has relatively siloed working groups feeding independently into decision making.
Wider data sharing
The information used to build, refine, and characterise models of infectious disease might include raw data on the spread of the disease (numbers of cases, hospital admissions, deaths, etc), data on the parameters that feed into models (transmissibility, severity, incubation period, etc), assumptions underlying model structure (is there a long pre-infectious “exposed period” etc), and the outputs of models (predictions of case numbers, hospital admission, etc). Retrospectively, some SPI-M members identified a problem with data accessibility (particularly for raw data and parameters).
Some groups had access to better quality data that were not shared with all modelling groups. In a BBC documentary, “Lockdown 1.0, following the science,” broadcast in November 2020, the chair of SPI-M said that differences in data availability were “inevitable” and that some groups would necessarily have a “head start” because of the effort they had put in to create the networks through which the data were being shared. However, some researchers on SPI-M were forced to resort to dredging Wikipedia early on in the pandemic, as it was the only data stream that was publicly available at the time.38 Some modellers described the data that were publicly available as being of extremely poor quality.
Initially, there was only limited data sharing across countries, reducing the learning possible from others’ earlier experience. The importance of international data sharing has been shown repeatedly. A good is example is the dissemination of genomic data on new SARS-CoV-2 variants through the Global Initiative on Sharing Avian Influenza Data (GISAID).39
The initial lack of data sharing could have contributed to mistakes made early on in the pandemic. Groups with access to poorer quality data did not feel able to challenge the conclusions of groups with access to better quality data, leading to poor modelling outcomes. In March 2020, for example, SPI-M overestimated the doubling time of the UK epidemic. Although some of the modelling groups were generating more accurate values, their estimates did not find their way to SAGE. Instead, an overestimated doubling time of 5-7 days appears in the SAGE minutes of 18 March.40 This incorrect figure is the one that was provided to policy makers. The true doubling time was more likely to be around three days (as estimated by minutes from 25 March 2020, a week later).57As a result of the early incorrect doubling time, Patrick Vallance, the chief scientific adviser, would claim we were “maybe four weeks or so behind [Italy] in terms of the scale of the outbreak” when in fact the UK was more like two weeks behind.4142 This incorrect calculation may have provided a false sense of complacency and contributed to the UK’s significant delay in taking measures to suppress the pandemic,43 which resulted in the avoidable loss of tens of thousands of lives.4445
SPI-M have since instituted more robust methods of model averaging. These were used, for example, to come up with consensus views on estimates of the reproduction number and growth rates of the disease. However, it is not clear that problems with sharing of other data sources required to construct effective models have been resolved (for instance individual-level data on infections, hospital admissions, and deaths; international data). More comprehensive and timely sharing of other data sources might reduce uncertainty and increase accuracy in models, improving their usefulness. Models that have different structures and both more relevant and more accurate parameters would also reduce the effect of structural uncertainty (box 1).
Communicating the modelling
Open and clear communication of the outputs of disease transmission models (and the entire modelling process) is vital to support policy decisions and increase the public’s understanding of, and desire to abide by, rules that are informed by such models. Indeed, one of the main criticisms surrounding mathematical modelling during the pandemic has been the lack of clear and consistent communication.46 Unfortunately, outputs of complex models do not speak for themselves—they need to be explained. This does not necessarily mean that modellers should advocate for specific policies, but they do need to explain what the models can and can’t be used for, and why. Some SPI-M scientists recognised the importance of public communication but understandably said that they did not have the time to engage fully, given that their energies were devoted to refining and running models.33
As a simple example, poor public understanding of exponential growth has been shown to hinder implementation of effective strategies to control infectious disease.47 The lower the levels of understanding of exponential growth, the lower the levels of compliance with anti-covid measures, including the use of face coverings and social distancing. People who find it hard to accurately estimate the speed of disease spread also find it difficult to see the importance of disease control mitigations and are less likely to implement or observe them.
That said, communication must also involve listening—and different people listen differently and from different perspectives. An analysis of public perceptions of UK Test and Trace and its implications for disease modelling concludes that modellers having better understanding of public perceptions of Test and Trace could have changed the structure and parameter ranges of the models for the better.27 Meanwhile, policy makers might use modelling results to support pre-existing policy goals—a kind of policy based evidence selection rather than true evidence based policy.42 Policy makers also do not always engage with or understand the process underlying the modelling results they choose to base policy on. They may see the complexity of the modelling process as giving the model outputs an “illusion of certainty,” making it difficult to question the results.48 Excellent communication is necessary but not sufficient for models to appropriately inform policy.
Pandemic policy making differs from normal time policy making in several ways. Firstly, the need for rapid action leaves less time for a proper assessment of the available evidence, adding uncertainty to the modelling and making it hard to communicate the nuances behind modelling results to policy makers. Conversely, the high visibility of much of the scientific evidence during the covid pandemic may have meant policy makers felt under increased public scrutiny and therefore under greater pressure to make evidence based decisions.
Another challenge is that the lack of context surrounding model results means they are open to misinterpretation by the media or exploitation. Good modelling practice is to present a range of scenarios for different combinations of parameter values alongside prediction intervals, which can help to express uncertainty. In particular, the development of reasonable worst case scenarios follows the public health modelling mantra “plan for the worst, but hope for the best.” These worst case scenarios often generate the most startling projections and consequently capture the news headlines. Particularly if policy action is taken to avoid the worst outcomes, this can lead to accusations of doom mongering and distrust in future model predictions when these scenarios do not then play out in reality.
A third problem arising from inadequate communication surrounding official modelling is that it leaves a media vacuum, which will necessarily be filled by other academic or amateur modelling efforts. Although there is certainly room for different modelling perspectives, those modellers who present their findings in the most media friendly manner tend to dominate the public perception of modelling. For example, just over a week after the Imperial College modelling group published report 9, a group of modellers at the University of Oxford set out their results in a preprint.49 Using a simplistic model, they proposed that the UK’s epidemic has “already led to the accumulation of significant levels of herd immunity.” The article was distributed to the media through a commercial public relations company. Unusually for academic papers, the same PR firm was the only contact listed on the preprint.
As a result of their successful media strategy, the “Oxford model” was presented with the same credibility as the Imperial model,50 despite the modelling being of very different quality. Although many scientists openly challenged the headline results from the Oxford model,51 their voices were largely drowned out in the media furore. Even without the official sanction of peer review, the media coverage catapulted the authors to a prominent position from which they were able to influence government policy. Their advice, which went directly to the top of government, may have influenced the decision to delay lockdown in the autumn of 2020.52
Communication of modelling is challenging at the best of times and made harder in a pandemic. But this does not mean modellers should not try. Ideally, the authoritative voice on the work should come from the modellers themselves. We need to train modellers to convey the nuances of the model results and their assumptions to a general audience—for example by producing lay summaries that they or well briefed intermediaries can use to engage with journalists to reduce the chances of misrepresentation.
This additional work of communication must be adequately resourced. Funding must be available for modelling teams to have the time, and access to the expertise, to undertake this communication, or for this communication to be undertaken by experts within government (such as the UK Health Security Agency or the Civil Service, both of which employ many excellent modellers) or an independent expert body such as the Royal Statistical Society or the Royal Society of Public Health. Expert communication must also be tailored to each audience and use an appropriate amount of detail (often less than modellers might wish). Decision makers should also receive basic training in how mathematical models inform policy, what questions to ask of modellers, and what the potential pitfalls are. Making explicit provision for communication is not an optional extra but a key part of maximising the benefit of modelling to inform policy and minimising the risk of misuse.
Finally, in order to sustain trust, modelling undertaken for the government should be made publicly available as soon as possible so that the results and the underlying assumptions of the models can be appropriately scrutinised. It is also important that modellers—alongside the interdisciplinary team assembled—should have input into the scenarios they choose to model. In particular, they should not feel restricted to model only those scenarios suggested to them by the government. Even if models are communicated perfectly, their use by other parties is not wholly (or often even largely) within the modellers’ control.
Risks of projecting too far ahead
Many SPI-M projections extended for four to six months53 and some for a year ahead or more.54 Fundamental shifts in the dynamics of the pandemic within that time frame can render the projections redundant, as we have seen several times with the emergence of new variants or changes in government policy. For instance, the projections in February 2021 that went up to April 2022, assumed no new variants and no vaccine waning.54 In fact, four new dominant variants have arisen since then (delta and omicron sub variants BA.1, BA.2 and BA.4/5) and vaccine waning has been an important factor in determining the trajectory of the pandemic and new vaccine policies.
Although delivering long term forecasts of what a pandemic might look like should not be prohibited, and SPI-M was transparent about the assumptions made (eg, no new variants emerging), the results of long term projections can nonetheless mislead because the likelihood of such fundamental shifts in pandemic dynamics is poorly understood by both policy makers and the public.
The problem in presenting projections over such a long timeframe is that they can instil a false sense of certainty and complacency, because they do not adequately acknowledge the likelihood (which has proved to be high with SARS-CoV-2) of such fundamental changes occurring. Moving to a shorter timeframe might also encourage policy makers to incorporate more uncertainty and anticipated reassessments into their plans and communication.
Other disciplines such as operational research or financial risk management have established methods that can incorporate the risk of rare, but potentially momentous events into decision making (eg, conditional value at risk strategies 55). One approach would be to incorporate these into the long term modelling framework. Another would be to use modelling timeframes of no longer than about four months. Of course, there may be modelling scenarios that are unlikely to be affected by trajectory changing events and for which longer time frames are suitable.
Epidemiological modelling is vital to understanding the current state of the pandemic and predicting what might happen in the future under different scenarios. Modelling has undoubtedly provided valuable input into the policies designed to tackle coronavirus, including the March 2020 lockdown. On the other hand, government has sometimes ignored modelling projections, such as when it decided not to impose stricter measures in September 2020, despite SAGE’s suggestions that doing so could halt the early exponential growth in cases.56
We have suggested some key questions for the public inquiry to ask about the input of modelling into government covid policy (box 1). With better communication, more openness to dialogue with other communities, and improved data sharing, epidemiological modelling could more successfully support the UK response to this and future pandemics.
Contributors and sources: CP has over 15 years of experience in developing mathematical and statistical models to inform policy. CAY’s research concentrates on combining different mathematical modelling methods to understand biological processes including epidemics. Both are members of Independent SAGE and have been active in helping explain mathematical models and their role in policy to the public during the pandemic. Both authors contributed equally and CP is guarantor.
Competing interests: We have read and understood BMJ policy on declaration of interests and declare we are both members of Independent SAGE and have been active in helping explain mathematical models and their role in policy to the public during the pandemic.
Provenance and peer review: Commissioned; externally peer reviewed.
This article is part of a series commissioned, peer reviewed, and edited by The BMJ. The advisory group for the series was chaired by Kara Hanson, and included Martin McKee, although he was not involved in the decision making on the papers that he co-authored. Kamran Abbasi was the lead editor for The BMJ.
This article is made freely available for personal use in accordance with BMJ's website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may download and print the article for any lawful, non-commercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained.https://bmj.com/coronavirus/usage