Role of mathematical modelling in future pandemic response policy

How mathematical modelling is used to inform policy

Mathematical modelling provides a framework in which we can formalise our assumptions about the processes we are trying to capture (eg, disease spread and impact), build them into a simplified representation of reality, and simulate forward in time to suggest what might happen in the future under different policy options.2223 Modelling is also extremely useful in understanding the underlying situation when we have incomplete or missing data,2425 and can shed light on what has happened in the past when the picture is murky, such as the effect of different public health mitigations.24A detailed review of how SPI-M was formed and how its work feeds into government policy via SAGE was published in 2021.2

Epidemiological modelling is more akin to science than to pure mathematics. The process involves iteratively building models, making predictions, comparing these predictions to observations, and then refining the models. Through this repetitive process modellers can build accurate, detailed, and robust representations of reality, which can then be used to speculate about what will happen in hitherto unseen scenarios. Most applications of mathematical modelling allow for many repeats of this cycle over periods of weeks, months, or even years. By contrast, synthesising appropriate data to populate and fine tune models in real time during a pandemic is an almost unique challenge in applied mathematics.2

Any modelling comes with various uncertainties and assumptions that need to be thought through, examined, and explained.26 Substantial errors in any area can derail the usefulness of the model, and, if not understood and recognised, cause harm. In the context of a rapidly evolving pandemic this is even more important. Good mathematical modelling must be transparent about all the sources of uncertainty (table 1) and provide sufficient detail to outsiders (including policy makers) to assess the model outputs.

SPI-M modelling during the pandemic has been admirably transparent about key assumptions and parameter estimates and has typically encompassed a range of scenarios. The models have incorporated inherent variability and highlighted many of the problems associated with unknown future events. Structural details of the SPI-M modelling are usually available in academic papers but are not easily accessible to a non-academic audience. Nonetheless, this transparency has not been sufficient to prevent mistakes or criticism. What then are the key questions around the role of modelling that the public inquiry should address?

Expert input was sometimes too narrow

As described above, model building is iterative. The structure of the model and its input parameters are continuously refined in light of the latest evidence and understanding about the dynamics of the disease and its spread. Perhaps the biggest threat to the usefulness of the models is when important information or knowledge relating to the dynamics is held by experts who are not connected to the modelling community, including the public.27 The modelling related to care homes during the covid-19 pandemic represents perhaps the most important cautionary tale.

Older and sicker populations were known early on to be at much higher risk of severe illness and death from covid-19. Modellers on SPI-M quickly understood that elderly people, and particularly those in care homes, were at high risk should they catch coronavirus. The need for protection of care home residents was also well appreciated, yet surprisingly the words “care homes” appear only twice in SAGE minutes during the first five months of the pandemic.28 Modellers were given access to the excellent hospital surveillance dataset at the start of the pandemic,29 but there was poor understanding of some of the important factors required for models to appropriately represent social care settings and thereby protect care homes. Experts on the care sector identified the intersecting factors of an extremely vulnerable population living in shared accommodation, frequent contact with friends and relatives in the community, the discharge of potentially sick patients from hospitals, the lack of personal protective equipment and low paid staff (with little access to sick pay, working across multiple homes as agency workers and more likely to live in multi-occupancy poor housing) as particular system vulnerabilities. Many of these issues, however, did not seem to be anticipated or explicitly taken into account by the modellers.

Although mathematical modellers could not be expected to have a prior understanding of the details of the social care sector and the interacting features of vulnerable populations and staff, they should have realised that they might be unaware of important factors and needed to seek relevant expertise. To social care experts, the vulnerabilities in the system were both knowable and known. However, modellers seem to have failed to identify the knowledge gap and so could not access relevant knowledge from those with the requisite expertise. Once the vulnerability of care homes became clearer to modellers, their specific features were successfully incorporated into models which then (albeit with hindsight) highlighted the high numbers of deaths if mitigations were not adequate.3031 In England and Wales there were more than 27 000 deaths in care homes during the first wave of the pandemic.32

Modelling subgroups convened by government should draw on as much diverse expertise as possible—an aspect also considered by the Parliamentary Committee on Science and Technology.33 Learning could be drawn from published literature on interdisciplinary working 34 in disaster response353637 and adapted to the UK situation. The mechanisms for ensuring interdisciplinary working must be in place and documented before a pandemic hits and should be agnostic to the nature of the pandemic or to the personal experience and networks of lead experts at the time. This recommendation also applies to the overall structure of SAGE, which has relatively siloed working groups feeding independently into decision making.

Wider data sharing

The information used to build, refine, and characterise models of infectious disease might include raw data on the spread of the disease (numbers of cases, hospital admissions, deaths, etc), data on the parameters that feed into models (transmissibility, severity, incubation period, etc), assumptions underlying model structure (is there a long pre-infectious “exposed period” etc), and the outputs of models (predictions of case numbers, hospital admission, etc). Retrospectively, some SPI-M members identified a problem with data accessibility (particularly for raw data and parameters).

Some groups had access to better quality data that were not shared with all modelling groups. In a BBC documentary, “Lockdown 1.0, following the science,” broadcast in November 2020, the chair of SPI-M said that differences in data availability were “inevitable” and that some groups would necessarily have a “head start” because of the effort they had put in to create the networks through which the data were being shared. However, some researchers on SPI-M were forced to resort to dredging Wikipedia early on in the pandemic, as it was the only data stream that was publicly available at the time.38 Some modellers described the data that were publicly available as being of extremely poor quality.

Initially, there was only limited data sharing across countries, reducing the learning possible from others’ earlier experience. The importance of international data sharing has been shown repeatedly. A good is example is the dissemination of genomic data on new SARS-CoV-2 variants through the Global Initiative on Sharing Avian Influenza Data (GISAID).39

The initial lack of data sharing could have contributed to mistakes made early on in the pandemic. Groups with access to poorer quality data did not feel able to challenge the conclusions of groups with access to better quality data, leading to poor modelling outcomes. In March 2020, for example, SPI-M overestimated the doubling time of the UK epidemic. Although some of the modelling groups were generating more accurate values, their estimates did not find their way to SAGE. Instead, an overestimated doubling time of 5-7 days appears in the SAGE minutes of 18 March.40 This incorrect figure is the one that was provided to policy makers. The true doubling time was more likely to be around three days (as estimated by minutes from 25 March 2020, a week later).57As a result of the early incorrect doubling time, Patrick Vallance, the chief scientific adviser, would claim we were “maybe four weeks or so behind [Italy] in terms of the scale of the outbreak” when in fact the UK was more like two weeks behind.4142 This incorrect calculation may have provided a false sense of complacency and contributed to the UK’s significant delay in taking measures to suppress the pandemic,43 which resulted in the avoidable loss of tens of thousands of lives.4445

SPI-M have since instituted more robust methods of model averaging. These were used, for example, to come up with consensus views on estimates of the reproduction number and growth rates of the disease. However, it is not clear that problems with sharing of other data sources required to construct effective models have been resolved (for instance individual-level data on infections, hospital admissions, and deaths; international data). More comprehensive and timely sharing of other data sources might reduce uncertainty and increase accuracy in models, improving their usefulness. Models that have different structures and both more relevant and more accurate parameters would also reduce the effect of structural uncertainty (box 1).

Communicating the modelling

Open and clear communication of the outputs of disease transmission models (and the entire modelling process) is vital to support policy decisions and increase the public’s understanding of, and desire to abide by, rules that are informed by such models. Indeed, one of the main criticisms surrounding mathematical modelling during the pandemic has been the lack of clear and consistent communication.46 Unfortunately, outputs of complex models do not speak for themselves—they need to be explained. This does not necessarily mean that modellers should advocate for specific policies, but they do need to explain what the models can and can’t be used for, and why. Some SPI-M scientists recognised the importance of public communication but understandably said that they did not have the time to engage fully, given that their energies were devoted to refining and running models.33

As a simple example, poor public understanding of exponential growth has been shown to hinder implementation of effective strategies to control infectious disease.47 The lower the levels of understanding of exponential growth, the lower the levels of compliance with anti-covid measures, including the use of face coverings and social distancing. People who find it hard to accurately estimate the speed of disease spread also find it difficult to see the importance of disease control mitigations and are less likely to implement or observe them.

That said, communication must also involve listening—and different people listen differently and from different perspectives. An analysis of public perceptions of UK Test and Trace and its implications for disease modelling concludes that modellers having better understanding of public perceptions of Test and Trace could have changed the structure and parameter ranges of the models for the better.27 Meanwhile, policy makers might use modelling results to support pre-existing policy goals—a kind of policy based evidence selection rather than true evidence based policy.42 Policy makers also do not always engage with or understand the process underlying the modelling results they choose to base policy on. They may see the complexity of the modelling process as giving the model outputs an “illusion of certainty,” making it difficult to question the results.48 Excellent communication is necessary but not sufficient for models to appropriately inform policy.

Pandemic policy making differs from normal time policy making in several ways. Firstly, the need for rapid action leaves less time for a proper assessment of the available evidence, adding uncertainty to the modelling and making it hard to communicate the nuances behind modelling results to policy makers. Conversely, the high visibility of much of the scientific evidence during the covid pandemic may have meant policy makers felt under increased public scrutiny and therefore under greater pressure to make evidence based decisions.

Another challenge is that the lack of context surrounding model results means they are open to misinterpretation by the media or exploitation. Good modelling practice is to present a range of scenarios for different combinations of parameter values alongside prediction intervals, which can help to express uncertainty. In particular, the development of reasonable worst case scenarios follows the public health modelling mantra “plan for the worst, but hope for the best.” These worst case scenarios often generate the most startling projections and consequently capture the news headlines. Particularly if policy action is taken to avoid the worst outcomes, this can lead to accusations of doom mongering and distrust in future model predictions when these scenarios do not then play out in reality.

A third problem arising from inadequate communication surrounding official modelling is that it leaves a media vacuum, which will necessarily be filled by other academic or amateur modelling efforts. Although there is certainly room for different modelling perspectives, those modellers who present their findings in the most media friendly manner tend to dominate the public perception of modelling. For example, just over a week after the Imperial College modelling group published report 9, a group of modellers at the University of Oxford set out their results in a preprint.49 Using a simplistic model, they proposed that the UK’s epidemic has “already led to the accumulation of significant levels of herd immunity.” The article was distributed to the media through a commercial public relations company. Unusually for academic papers, the same PR firm was the only contact listed on the preprint.

As a result of their successful media strategy, the “Oxford model” was presented with the same credibility as the Imperial model,50 despite the modelling being of very different quality. Although many scientists openly challenged the headline results from the Oxford model,51 their voices were largely drowned out in the media furore. Even without the official sanction of peer review, the media coverage catapulted the authors to a prominent position from which they were able to influence government policy. Their advice, which went directly to the top of government, may have influenced the decision to delay lockdown in the autumn of 2020.52

Communication of modelling is challenging at the best of times and made harder in a pandemic. But this does not mean modellers should not try. Ideally, the authoritative voice on the work should come from the modellers themselves. We need to train modellers to convey the nuances of the model results and their assumptions to a general audience—for example by producing lay summaries that they or well briefed intermediaries can use to engage with journalists to reduce the chances of misrepresentation.

This additional work of communication must be adequately resourced. Funding must be available for modelling teams to have the time, and access to the expertise, to undertake this communication, or for this communication to be undertaken by experts within government (such as the UK Health Security Agency or the Civil Service, both of which employ many excellent modellers) or an independent expert body such as the Royal Statistical Society or the Royal Society of Public Health. Expert communication must also be tailored to each audience and use an appropriate amount of detail (often less than modellers might wish). Decision makers should also receive basic training in how mathematical models inform policy, what questions to ask of modellers, and what the potential pitfalls are. Making explicit provision for communication is not an optional extra but a key part of maximising the benefit of modelling to inform policy and minimising the risk of misuse.

Finally, in order to sustain trust, modelling undertaken for the government should be made publicly available as soon as possible so that the results and the underlying assumptions of the models can be appropriately scrutinised. It is also important that modellers—alongside the interdisciplinary team assembled—should have input into the scenarios they choose to model. In particular, they should not feel restricted to model only those scenarios suggested to them by the government. Even if models are communicated perfectly, their use by other parties is not wholly (or often even largely) within the modellers’ control.

Risks of projecting too far ahead

Many SPI-M projections extended for four to six months53 and some for a year ahead or more.54 Fundamental shifts in the dynamics of the pandemic within that time frame can render the projections redundant, as we have seen several times with the emergence of new variants or changes in government policy. For instance, the projections in February 2021 that went up to April 2022, assumed no new variants and no vaccine waning.54 In fact, four new dominant variants have arisen since then (delta and omicron sub variants BA.1, BA.2 and BA.4/5) and vaccine waning has been an important factor in determining the trajectory of the pandemic and new vaccine policies.

Although delivering long term forecasts of what a pandemic might look like should not be prohibited, and SPI-M was transparent about the assumptions made (eg, no new variants emerging), the results of long term projections can nonetheless mislead because the likelihood of such fundamental shifts in pandemic dynamics is poorly understood by both policy makers and the public.

The problem in presenting projections over such a long timeframe is that they can instil a false sense of certainty and complacency, because they do not adequately acknowledge the likelihood (which has proved to be high with SARS-CoV-2) of such fundamental changes occurring. Moving to a shorter timeframe might also encourage policy makers to incorporate more uncertainty and anticipated reassessments into their plans and communication.

Other disciplines such as operational research or financial risk management have established methods that can incorporate the risk of rare, but potentially momentous events into decision making (eg, conditional value at risk strategies 55). One approach would be to incorporate these into the long term modelling framework. Another would be to use modelling timeframes of no longer than about four months. Of course, there may be modelling scenarios that are unlikely to be affected by trajectory changing events and for which longer time frames are suitable.