SARS-CoV-2 antigen lateral flow tests for detecting infectious people: linked data analysis
BMJ 2022; 376 doi: https://doi.org/10.1136/bmj-2021-066871 (Published 23 February 2022) Cite this as: BMJ 2022;376:e066871Linked Opinion
Self-testing for asymptomatic non-contacts using rapid antigen tests
- Jonathan J Deeks
, professor of biostatistics12, - Anika Singanayagam, clinical research fellow34,
- Hamish Houston, clinical research fellow3,
- Alice J Sitch, associate professor of biostatistics12,
- Seran Hakki, research associate3,
- Jake Dunning, senior clinical research fellow5,
- Ajit Lalvani, chair in infectious disease3
- 1Test Evaluation Research Group, Institute of Applied Health Research, University of Birmingham, Birmingham B15 2TT, UK
- 2NIHR Birmingham Biomedical Research Centre, University Hospitals Birmingham NHS Foundation Trust, UK and University of Birmingham, UK
- 3NIHR Health Protection Research Unit in Respiratory Infections, National Heart and Lung Institute, Imperial College London, London, UK
- 4Department of Infectious Diseases, Imperial College London, London, UK
- 5NIHR Health Protection Research Unit in Emerging and Zoonotic Infections, Centre for Tropical Medicine, University of Oxford, Oxford, UK
- Correspondence to: J J Deeks j.deeks{at}bham.ac.uk
- Accepted 2 February 2022
Abstract
Objectives To investigate the proportion of lateral flow tests (LFTs) that produce negative results in those with a high risk of infectiousness from SARS-CoV-2, to investigate the impact of the stage and severity of disease, and to compare predictions made by influential mathematical models with findings of empirical studies.
Design Linked data analysis combining empirical evidence of the accuracy of the Innova LFT, the probability of positive viral culture or transmission to secondary cases, and the distribution of viral loads of SARS-CoV-2 in individuals in different settings.
Setting Testing of individuals with symptoms attending NHS Test-and-Trace centres across the UK, residents without symptoms attending municipal mass testing centres in Liverpool, and students without symptoms screened at the University of Birmingham.
Participants Evidence for the sensitivity of the Innova LFT, based on 70 individuals with SARS-CoV-2 and LFT results. Infectiousness was based on viral culture rates on 246 samples (176 people with SARS-CoV-2) and secondary cases among 2 474 066 contacts; distributions of cycle threshold (Ct) values from 231 497 index individuals attending NHS Test-and-Trace centres; 70 people with SARS-CoV-2 detected in Liverpool and 62 people with SARS-CoV-2 in Birmingham (54 imputed).
Main outcome measures The predicted proportions who were missed by LFT and viral culture positive and missed by LFT and sources of secondary cases, in each of the three settings. Predictions were compared with those made by mathematical models.
Results The analysis predicted that of those with a viral culture positive result, Innova would miss 20% attending an NHS Test-and-Trace centre, 29% without symptoms attending municipal mass testing, and 81% attending university screen testing without symptoms, along with 38%, 47%, and 90% of sources of secondary cases. In comparison, two mathematical models underestimated the numbers of missed infectious individuals (8%, 10%, and 32% in the three settings for one model, whereas the assumptions from the second model made it impossible to miss an infectious individual). Owing to the paucity of usable data, the inputs to the analyses are from limited sources.
Conclusions The proportion of infectious people with SARS-CoV-2 missed by LFTs is substantial enough to be of clinical importance. The proportion missed varied between settings because of different viral load distributions and is likely to be highest in those without symptoms. Key models have substantially overestimated the sensitivity of LFTs compared with empirical data. An urgent need exists for additional robust well designed and reported empirical studies from intended use settings to inform evidence based policy.
Introduction
The use of lateral flow tests (LFTs) to detect SARS-CoV-2 has proliferated worldwide.1 In the UK, twice weekly testing is encouraged in people without symptoms and is currently freely available to all (test to detect).2 One-off use has been recommended before visiting care home residents and attending social gatherings (test to protect)3 and large scale events (test to enable).45 LFTs were recently (until February 2022) used for vaccinated travellers returning to the UK, and they are also used to end self-isolation after a positive test result (test to release). LFTs are cheap and can quickly provide results, limiting delays to self-isolation, which is known to reduce transmission.6 Although empirical data show LFTs give a positive result when virus is present on a swab in high quantities,7 and therefore can detect people who are likely to be infectious, the proportion missed who are infectious has not been evaluated.
As dedicated studies of LFT performance in specific applications are scarce, policy decisions on the use of LFTs have largely been supported by mathematical models. However, key assumptions have been made about the relationships between the distribution of viral load, estimates of diagnostic sensitivity, and measures of infectiousness.8910 Simplistic approaches have been adopted where it is assumed that those with viral loads above a fixed threshold (often assumed to be 106 copies/mL)) are highly likely to be infectious, whereas those with viral loads below the threshold are assumed to have little infectious virus.11 The same assumptions have been used to suggest that LFT sensitivity estimates should be recalibrated when considering the detection of infectious disease.12 These assumptions, which so far have rarely been based on empirical data analysis, prompt claims that LFTs identify many currently infectious individuals and only give a negative result during the pre-infectious or post-infectious phases.181314 To date, only one study has used empirical data to produce these predictions.15
As the proposed use of LFTs for test to enable, test to protect, and test to release requires false negative results to be minimised,16 an assessment of the current evidence base to understand the sensitivity of LFTs for detecting infectious people within such applications is critical. Without this understanding, the resulting impacts on transmission cannot be accurately ascertained. Although LFTs are cheap, it will not be possible to perform formal cost effectiveness analyses of LFTs for specific community based applications until these impacts are better understood.
Difference in sensitivity between polymerase chain reaction and lateral flow tests
LFTs detect SARS-CoV-2 viral protein (antigen) and are less sensitive than reverse transcription polymerase chain reaction (RT-PCR) tests, which detect SARS-CoV-2 genetic material (RNA).7 Test sensitivity is related to the amount of virus in a sample, with the limit of detection being the lowest amount that a test can accurately detect. Although direct comparisons have been limited, the lowest amount of virus detectable by LFTs and RT-PCR tests has been shown to differ by several orders of magnitude,171819202122 and one study suggests RT-PCR tests can detect SARS-CoV-2 earlier than LFTs.23 Lower limits of detection are often quantified in serial dilution studies. In these studies, there is no difference between the virus that LFTs detect and the virus that LFTs miss in its capacity to infect human cells—the issue is simply whether the quantity of virus in the sample is sufficient to register on the test. An understanding of whether the lower limits of detection of LFTs are adequate to identify those who are infectious is important.
Challenges in defining a reference standard for infectiousness
Evaluating the accuracy of a test requires a target condition (the disease state to detect) and a reference standard to establish whether an individual does or does not have the target condition.2425 In the case of SARS-CoV-2, there is no agreed target condition for infectiousness and no reference standard that accurately distinguishes current active infection from recent infection at the time of testing.16 The results of RT-PCR tests can remain positive for an average of 17 days after initial infection,26 including a period after active infection has subsided (a post-infectious period). Two quantities likely to be related to both current active infection and infectiousness at the time of testing (but too inaccurate to be considered a reference standard) are direct evidence of transmission to secondary cases (ie, those who acquired the virus from an infected individual) and the ability to culture virus in laboratory cell lines, aligning with the Medicines and Healthcare Products Regulatory Agency definition of a relevant target condition of “current active infection” as “an infection in which the causative organism has the potential, either now or in the future, to cause disease or onward transmission.”27
Influential models, however, have focused on a narrower target condition of identifying individuals who are supposedly infectious at the time of testing.8 In some models, infectiousness and the sensitivity of LFTs are both defined using viral load thresholds, which can mean it is highly unlikely (or even impossible) for LFTs to miss an individual who is infectious. For example, a model might assume the sensitivity of LFTs to be 100% below a given viral load and define all people who are infectious to be below this same threshold, meaning no infected individual can be missed.
Estimating how reliably LFTs detect infectiousness
Just as LFT sensitivity depends on viral load, the probabilities of a positive viral culture result and secondary transmission are also related to viral load. Viral load can be measured from upper respiratory tract swabs using a quantitative RT-PCR with appropriate standard curves, but often is inferred semiquantitatively using cycle threshold (Ct) values from diagnostic laboratory RT-PCR assays, which report the number of reaction cycles required before virus is detectable (lower values indicating higher viral load).
In this article we link available empirical data describing the viral load (Ct value) distributions from populations in three intended use settings for LFTs, and the relationships between viral load and LFT sensitivity, probability of a positive viral culture result, and probability of secondary transmission. We calculate estimates of how well a LFT used in these settings would detect individuals with features that make them at high risk of being infectious, compare our empirically based predictions with the assumptions and outputs of two influential models,89 and investigate how future tests that can detect SARS-CoV-2 at lower viral loads would identify those who are likely to be infectious.
Methods
Inclusion criteria, selection, and extraction
We focus on the Innova assay (widely distributed in the UK) and consider its application in UK test settings. We undertook repeated searches and surveillance of preprints and unpublished reports of studies of LFTs for the Cochrane review of rapid tests for SARS-CoV-2 (see supplementary material for details of search strategies and studies7). From these searches we identified studies that reported the accuracy of Innova in real world settings stratified by viral load or Ct value, together with distributions of cases by viral load or Ct value in settings where the test was applied. From the search and citation tracking of articles in the specialty, we identified studies that reported on the relationship of viral load with viral culture and secondary attack rates. We aimed to identify studies of these relationships with sample sizes large enough to provide estimates of the risk of culture and transmission across the full range of viral loads. We also required availability of equations to convert Ct values to viral loads to harmonise data from different RT-PCR machines.
Two authors (JJD and AJS) independently extracted data from published or preprint versions of papers, with discrepancies resolved by consensus. Data from figures were digitised (Quintessa, Graph Grabber Version 2.0.2) and means of multiple duplicate extractions used.
The supplementary material provides full details of the equations and methods used. For ease of standardisation given the data available, we elected to express viral loads in terms of Ct values produced by the RT-PCR assay used by the Lighthouse laboratories. On this scale a viral load of 106 viral copies/mL corresponds to a Ct value of 18.3, 105 viral copies/mL to a Ct value of 21.3, and 104 viral copies/mL to a Ct value of 24.4.28 We produced equations that allowed us to predict the sensitivity of the Innova assay, probability of viral culture success, and probability that a case would infect a secondary case based on Ct value.
Predicting the proportion of people who would test negative on LFT but positive for viral culture and transmission
Our focus was to identify the joint probability that people are likely to be infectious (in that they have a viral culture positive result or are a secondary case) and that they test negative on Innova. At each Ct value, the probability that an individual with a positive culture result is missed is obtained by multiplying the probability of culture positivity at that Ct value by 1−sensitivity of the test at that Ct value. Similarly, the probability that an infected individual is missed who is the source of a secondary case is obtained by multiplying the probability of being a secondary case by 1−sensitivity of the test at that Ct value.
These predicted probabilities were applied to distributions of Ct values from field studies of Innova in intended use settings to predict the total percentage of missed infected individuals who would be detectable on RT-PCR, viral culture positive, and a source of secondary cases. Estimates of the total percentages of people in each setting who would be viral culture positive and a source of secondary cases were obtained in a similar way.
Comparison with previous models
Two influential models cited in testing policy documents (box 1) are considered—one from the UK9 and one from the US.8 We applied the functions for sensitivity and infectiousness used in these models to the distributions of Ct values from the same field studies of Innova in intended use settings, and estimated the percentage of people with viral culture positive and false negative results on the LFT and a source of transmission and false negative on the LFT, and compared these results with the empirical data.
Sources of data
Liverpool: sensitivity of Innova in asymptomatic mass testing, and distribution of Ct values28
5869 Liverpool residents were dual tested with Innova and reverse transcription polymerase chain reaction (RT-PCR) undertaken at a Lighthouse laboratory, 74 of whom had a positive test result on RT-PCR and 70 of those had a non-void LFT result. Figure 1 shows the numbers tested and the sensitivity of Innova, by Ct value. This pilot was the largest evaluation of an LFT with mass testing by non-healthcare workers, and it provides results most likely to be applicable to current implementation.
Public Health England, Colindale: relationship of Ct values and viral culture29
Virus culture on vero E6 cells was attempted in 324 upper respiratory tract samples from 253 infected people who tested positive for SARS-CoV-2 by RT-PCR. Virus was isolated from 133 (41%) of the samples (n=111 infected people) as presented in figure 1, grouped by Ct value. The distribution of viral culture positivity closely agrees with that observed in a larger set from Marseille.30 We preferred to use the Public Health England data owing to the availability of a calibration study from this laboratory that was used to convert Ct values into viral copy numbers, allowing comparison with data from the Lighthouse laboratory assay (personal communication, 2021). A caveat is that this calibration study was performed using a standard curve using Orf1ab RNA, in the absence of an RdRp equivalent. Further work (unpublished) from the laboratory has shown that Orf1ab and RdRp Ct values are similar from the same samples, with additional analysis performed to assess for effects of differences in assay format. Therefore, despite the caveats discussed, we consider these to be the best available data at the present time.
NHS Test and Trace: relationship of viral load and transmission with distribution of Ct values1531
Data were analysed from the NHS Test-and-Trace system to investigate the probability of contacts of known infected people who were tested using the Lighthouse laboratory becoming infected. Contacts were identified who had a RT-PCR test within 10 days of the index case and were named only by a single case. Of 2 474 066 eligible contacts, 231 498 (9.4%) tested positive by RT-PCR. Logistic regression was used to estimate the relationship between RT-PCR confirmed infection in contacts and the index case’s Ct value using splines to account for non-linearity in continuous variables. The fitted line was digitally extracted from the paper and a simple logistic function was found adequate to regenerate it. An earlier version of the manuscript included a histogram providing the distribution of Ct values in index cases.15
University of Birmingham student testing; distribution of Ct values10
Before they returned home at the end of 2020, more than 7000 students at the University of Birmingham were tested once by trained non-healthcare workers using Innova. All those who tested positive, and a 10% sample of those who tested negative, were verified by RT-PCR in the university NHS Turnkey laboratory using the same assay as the Lighthouse laboratories. Ct values for the two true positives and six false negatives were available. As only 10% of the students with a negative result with Innova were tested, Ct values for false negatives were imputed (nine for every one observed) with three at the observed value, two each ±1 Ct value, and one each ±2 Ct values compared with the observed.
Mathematical models of infectiousness89
Quilty et al (the Edmunds model) used an agent based model to simulate the viral load dynamics of exposed contacts each day and predicted onward transmission under different quarantine and testing strategies.9 The authors included testing using either RT-PCR or LFTs. The model assumed a four level function of sensitivity for LFTs across Ct values, and that individuals were infectious if Ct values were <30. The model predicted that releasing individuals from quarantine if they had a negative LFT result on day 7 after exposure, or daily testing using a LFT without quarantine, would create reductions in transmission equivalent to 14 days of post-exposure quarantine.
Larremore et al (the Mina model) used compartmental and agent based models to simulate the viral load dynamics of infected individuals each day, and considered how tests with lower levels of detection of 105 viral copies/mL and 103 viral copies/mL performed in detecting infectious individuals.8 Participants with viral loads >106 viral copies/mL were considered infectious. Comparisons were made between the two tests, and between different testing frequencies. Little difference was found between the tests, whereas more frequent testing led to larger reductions in the time spent infectious and not self-isolating.
Simulating the impact of LFTs with lower limits of detection
We simulated tests able to detect lower levels of virus by translating the curve for the sensitivity of the Innova LFT by changes equivalent to 101-fold, 102-fold, 103-fold, and 104-fold lowering of the lower limit of detection (each step equivalent to a Ct change of 3.05). This model assumes that at each step the sensitivity of the test improves at each Ct value to be equivalent to that of the Ct value minus 3.05 for the previous test in the sequence.
Patient and public involvement
No patients or members of the public were involved in the design, conduct, reporting, or dissemination plans of our research. Patient and public involvement will have contributed to the primary studies that are used in our linked data analysis. Our linked data analysis required specialist input; it was not possible to provide training to enable patient and public involvement. We were aware from engagement of the general public that further information on the ability of LFTs to detect infectious people with SARS-CoV-2 was desired.
Results
Viral load distribution from Innova field studies in intended use settings
Datasets were obtained for Ct values for testing in three settings: symptomatic testing at an NHS Test-and-Trace centre,15 mass testing in Liverpool in residents without symptoms,28 and in students at the University of Birmingham32 (see box 1).
Sensitivity of Innova
Two evaluations of the Innova LFT in real world settings by non-healthcare workers were identified that presented findings stratified by Ct value that could be converted to viral load (see supplementary material). Data from mass testing in Liverpool28 were used for sensitivity (70 RT-PCR positive people with non-void LFT results) over data from the University of Birmingham32 (eight RT-PCR positive people) based on sample size (see box 1). The top panel in figure 1 shows the reported proportion detected (with 95% confidence intervals) in each of four Ct categories, with the fitted curve allowing prediction of sensitivity at each Ct value. The sensitivity of Innova was predicted to be >95% at Ct values ≤14 (2.6×107 viral copies/mL) but <50% at Ct values ≥22 (6.1×104 viral copies/mL).
Sensitivity of Innova lateral flow test; probabilities of positive viral culture and secondary transmission; and probabilities of positive viral culture and Innova negative and source of secondary transmission and Innova negative by cycle threshold (Ct) value. Empirical evidence sources for sensitivity on Innova,28 probability of positive viral culture,29 and risk of secondary transmission31
Viral culture and secondary transmission rates
Similar risks have been shown across the full range of viral loads in viral culture samples from two large cohorts2930: the Public Health England Colindale cohort was used because a conversion equation from the laboratory enabled us to approximate Ct values to viral loads (personal communication, 2021) (see box 1).29 The middle panel in figure 1 shows the reported proportion who were culture positive (with 95% confidence intervals) in each of the five Ct categories reported, with the fitted curve allowing prediction of culture positive rates at each Ct value. Viral culture rates varied from 95% at Ct values <11 to <10% at Ct values >26.27
Two studies were identified that reported secondary transmission rates,3133 but only the NHS Test-and-Trace data provided estimates at lower viral loads (see box 1). As no raw data were reported, the risk relationship was extracted from the fitted curve. The middle panel in figure 1 shows the relationship between Ct value and secondary transmission, with rates declining from 11% at Ct values of 15, to 4% at Ct values of 29.34 Importantly, no step change is discernible related to a minimal level of virus for positive viral culture or transmission.
How sensitive is Innova for detecting infectious people?
The bottom panel in figure 1 shows at each Ct value the joint probability that an individual would be culture positive or a source of secondary transmission and would have a false negative result on the Innova LFT. The probabilities are low at low Ct values (when infected people will be successfully detected by the test) but rise as Ct values increase (and the sensitivity of the test decreases). The probability of an infected individual being missed and viral culture positive is >10% at Ct values between 18 and 25; and the probability of being missed and a source of transmission is >3% for Ct values between 22 and 32. At higher Ct values the probabilities of being viral culture positive or a source of onwards transmission are low. Thus although many infected people with high Ct values will be missed, they are unlikely to be viral culture positive or transmit infection to others.
What is the impact of different applications on the ability of LFTs to detect infectious people?
The distribution of viral loads (Ct values) determines the impact of the risks in figure 1 (bottom panel) on the effectiveness of any public health strategy where LFTs have been applied. Figure 2 shows the distributions of Ct values in three settings overlaid with the predicted risks of missing people with a positive viral culture result (figures in left panel) and transmission (figures in right panel).
Distributions of cycle threshold (Ct) values of individuals positive for SARS-CoV-2 in different settings. Birmingham University used imputation owing to proportional sampling design. *Distribution from Lee et al 2021.15 †Distribution from García-Fiñana et al 2021.28 ‡Distribution from Ferguson et al 202132
Ct values in people with presumed symptoms from the NHS Test-and-Trace programme (fig 2, top two panels) showed higher viral loads (lower Ct values) than other settings. The analysis predicts that the following would be missed by the Innova LFT: two fifths of those with SARS-CoV-2 detectable on RT-PCR (38%), one fifth (20%) with a positive viral culture result, and two fifths (38%) who are a source of transmission (table 1).
Predicted sensitivity of Innova lateral flow test to detect people with a positive viral culture result and who are a source of transmission compared with predictions from models. Values are percentages
In the other two settings, people without symptoms were tested and viral loads were lower (ie, higher Ct values) (fig 2, middle and bottom panels). The percentages of people predicted to be viral culture positive or sources of transmission were lower, as expected, but the ability of the test to identify them correctly also decreased. In Liverpool and Birmingham, the analysis predicts that the Innova LFT would miss 59% and 92% of those with a positive RT-PCR result, respectively, 29% and 81% of those with a positive viral culture result, and 47% and 90% who are a source of transmission.
How do the predictions of test sensitivity compare with observed sensitivity?
The predicted proportion of infected people detectable by RT-PCR match closely with the observed sensitivity data from the three settings. The observed sensitivity of the Innova LFT from the Porton Down/University of Oxford study was 58% (95% confidence interval 52% to 63%)22 (when tested by non-healthcare workers) compared with a prediction of 62%; in Liverpool 40% (28% to 52%) were detected28 compared with a prediction of 41%, and in Birmingham 3% (1% to 16%) were detected32 compared with a prediction of 8%. This close similarity between observed and predicted sensitivity illustrates that the observed differences between settings are likely largely driven by differences in the distribution of viral load, which in turn relates to the clinical disease spectrum and the timepoint in the course of infection when individuals are tested.
How do these estimates compare with previous models?
We consider the two key models: a UK model from the Edmunds group described in Quilty et al9 and a US model from the Mina group described in Larremore et al8 (see box 1), which have been cited in government reports and policy documents. Both models simulate a viral load for individuals on each day (created by modelling a trajectory of viral loads over time) and classify individuals as infectious during times when viral loads are above a chosen value indicative of infectiousness and classify them as detected when they are above a chosen value relating to the lower level of detection of the test. The models use probabilistic functions for infectiousness and sensitivity at viral loads above these lower limits (fig 3). The findings are strongly influenced by whether the lower limit at which individuals are deemed infectious is higher or lower than the lower limit at which the test detects infectiousness.
Comparison of model assumptions of sensitivity and infectiousness (or defined as infectious by model) with empirical data. The UK (Edmunds) model is described in Quilty et al 20219 and the US (Mina) model is described in Larremore et al 2021.8 Empirical evidence sources for sensitivity on the Innova lateral flow test,28 probability of positive viral culture,29 and risk of secondary transmission31
The UK model cites laboratory studies of the Innova LFT as the source for its sensitivity function.22 The top panel in figure 3 shows that the UK model’s four level function overestimates the sensitivity of the test compared with the real world sensitivity data from the Liverpool mass testing pilot.28 The US model used a simplistic single step function for sensitivity, from 0% to 100% at either 100 000 viral copies/mL (Ct=22.3, shown) or in a sensitivity analysis at 1000 viral copies/mL (Ct=33.6, not shown), both overestimating sensitivity at high viral loads and underestimating at low viral loads.
The UK model cites the same source for its infectiousness function as used in our empirical analysis for probability of viral culture,29 but modelled infectiousness using a constant risk for high viral loads below the Ct value of 30, and with zero risk at lower viral loads at and above a Ct value of 30 (fig 3, middle panel). This consideration of infectiousness as a binary yes or no concept related to viral load is contrary to the empirical evidence cited. The US model assumed a lower limit of infectiousness at 1 000 000 viral copies/mL (Ct=18.3), above the lower limit of detection of the test, and modelled the increase in infectiousness as linear (shown), exponential, and step functions with Ct values (sensitivity analyses, not shown).
The bottom panel in figure 3 shows the risk of people being both infectious and having a false negative result at each Ct value for both models. The UK model shows a risk of 5% at low Ct values, with an increased risk only for Ct values between 27 and 29.9 and no risk at higher values. Applying this risk function predicts much lower risks of infectious people having a false negative result than the empirical data (8%, 10%, and 32% in NHS Test-and-Trace, Liverpool mass testing, and University of Birmingham settings (table 1).
The US model assumed a lower limit for infectiousness above the level of viral load where tests were assumed to have 100% sensitivity, creating an extreme scenario where few infectious people are ever predicted and none are ever missed (fig 3, bottom panel). This feature does not change across the sensitivity analyses considered in the paper. This combination of assumptions means the model is destined to estimate sensitivity for infectiousness as 100% in all settings (table 1). No differences in sensitivity between tests considered in the paper with lower limits of detection of 103 and 105 are observed, as both have 100% sensitivity at viral loads <106 viral copies/mL lower limit of infectiousness. Consequently, the authors’ conclusion that the low sensitivity of the tests does not materially affect their suitability for use in mass testing8 created by this combination of unrealistic assumptions, is misleading. The small difference between tests with different lower levels of detection in the authors’ analysis results from the time point when infections are first detected being earlier with more sensitive RT-PCR tests when used in a daily schedule.
How much more sensitive do LFTs need to be to detect infectious people?
We simulated tests able to detect increasingly lower levels of virus by translating the curve for the sensitivity from the top panel in figure 1 to reflect increases equivalent to detection of 10-fold, 100-fold, 1000-fold, and 10 000-fold lower concentrations of viral copies/mL (each step corresponding to a 3.05 decrease in Ct values2831). Substantial increases in detection of infected people predicted to have viral culture positive results or to be sources of transmission are seen with more sensitive tests (table 2). In the NHS Test-and-Trace and Liverpool mass testing settings, the Medicines and Healthcare Products Regulatory Agency’s guidance on a minimum sensitivity of 97% to rule-out infection27 is met by tests able to detect viral loads 1000 to 10 000 times lower than the current performance of the Innova LFT, which might not be achievable, or might require a combination of tests.
Predicted sensitivity of lateral flow tests with lower limits of detection to identify people with RT-PCR detectable results, viral culture positive results, or to be sources of transmission. Values are percentages
Discussion
Principal findings and comparison with other studies
LFTs are an important tool in controlling the covid-19 pandemic because they are cheap, provide results quickly, and are widely available, reducing delays to self-isolation when a positive test result is detected. These public health benefits may outweigh the limited sensitivity of LFTs and might prove cost effective for certain applications. However, claims that LFTs can identify “the vast majority who are infectious” are likely overstated and risk providing false reassurance to those seeking to rule-out infection.2 Establishing how well tests detect current active SARS-CoV-2 infection or infectiousness is challenging as no accurate reference standard exists as a comparator. Instead, the ability of the test to identify those with established markers for infectiousness (namely positive viral culture result or sources of secondary cases) can be assessed. Using an analysis linking six sources of empirical evidence, we have shown that the current Innova LFT is predicted to miss substantial proportions of infectious individuals based on these established markers. The findings raise concerns about the utility and cost effectiveness of applying LFTs in test-to-enable and test-to-protect applications—further evaluation is required.
We have shown that infectious individuals are at most risk of having a false negative test result when their viral load is in the middle range of Ct values (between Ct values 18 and 25, we estimate a >10% probability that an individual with SARS-CoV-2 is missed by LFT and is also viral culture positive). The observed sensitivity of the test varies when used in different populations because of spectrum effects.3536 Thus, estimates of observed sensitivity for the test in one setting cannot be generalised to other settings, particularly when it is expected that the distributions of viral load will differ. The settings compared here show that differences can be large, with observed sensitivity being highest when average viral loads are high (eg, symptomatic testing) and low when average viral loads are lower (eg, testing asymptomatic students).
The impact of setting on performance shows that it is critical to monitor how LFTs perform in different applications to enable evidence based policy decisions. For example, monitoring LFT sensitivity in school children is urgently required37 as data show that viral kinetics might differ between children and adults, with viral loads potentially lower in children, which would put them at greater risk of false negative test results38 and lead to outbreaks in schools should false negative test results give false reassurance.
The UK criteria for RT-PCR testing of people with symptoms are narrow.39 Recent research, highlighted in UK media, has raised public awareness of the broader range of symptoms of covid-19,404142 and anecdotal evidence suggests widespread use of LFTs for symptomatic testing. Our findings raise concern that LFTs might provide false reassurance to those with unofficial symptoms, using the test result to rule-out infection.
Some models that claim LFTs work well as tests for infectiousness have assumed infectiousness only occurs above fixed lower bounds of viral load, above or close to the known lower limit of detection for LFTs.8 This is contrary to empirical evidence, which shows that viral culture and transmission—both indicators of infectiousness—have a continuum of risk, rather than step changes, and occur commonly at viral loads lower than those detectable by current LFTs.2931 Many papers that claim viral culture can only occur at very high viral loads have looked at only a few samples with low viral loads and thus cannot estimate these risks.4344 Going forward, it is essential that the assumptions made in models of test strategies are identified, scrutinised, and compared with empirical information, and that findings are confirmed in empirical studies.
Uncertainty
Our approach uses estimates from multiple sources to model the relationship between Ct value and the sensitivity, culture positivity, and secondary transmission associated with the Innova LFT (see supplementary material). Individual level data were preferred when available. When individual level data were not available, we obtained these data from graphics, or used the mid-point when aggregate data were given. All data were extracted in duplicate to ensure accuracy. There is the potential for error in the estimates from the logistic regression models fitted to describe the relationship between Ct value and sensitivity, culture positivity, and secondary transmission associated with the Innova LFT. After data extraction, we used modelling techniques to enable us to describe these relationships. Although model assumptions were checked, and, in some cases, multiple approaches were used, with the best performing carried forward, these models are based on assumptions to which no data will perfectly adhere. We cannot compute meaningful confidence intervals for these estimates as these results are calculated from combining the derived equations with the Ct distributions observed across the settings. Any sample size attributed to these results would be defined by us, rendering the confidence intervals meaningless. In our sensitivity analyses we assume lower levels of detection (better tests).
Limitations and key assumptions
Evaluating the accuracy of a test for current infection or infectiousness is challenging owing to the absence of a reference standard. All predictions are affected by the limitations of the markers of risk that are evaluated, and it is not possible to obtain precise estimates of the sensitivity of tests for these target conditions.
Both transmission and viral culture have limitations for estimating infectiousness, and neither are feasible to undertake in large studies of LFTs in asymptomatic groups. For example, while evidence of transmission confirms that an individual was infectious, absence of evidence does not indicate non-infectiousness. The likelihood of transmission depends highly on social, environmental, and behavioural factors, including when people have been isolating (preventing transmission that otherwise would have occurred). Moreover, obtaining clear evidence of transmission events is difficult, and assumptions are often made about who infected who, particularly at times of high prevalence when potential infectious contacts might be multiple. Transmission also does not identify the time point when an individual was infectious to contacts exposed for a duration of time, and so it might not be possible to pinpoint the exact infectious period.
People who are infectious must harbour viable virus (ie, it can replicate) that can be identified in cell culture of specimens. Nonetheless, the presence of viable virus in the upper respiratory tract is not sufficient for transmission to occur and therefore is not a perfect surrogate for infectiousness. To initiate infection in the next host the virus needs to be successfully emitted into the environment, retain its viability as it travels between hosts, and overcome an initial immune response. Importantly, viral culture requires technical expertise, and the sensitivity of cultures can be affected by sample handling (eg, transport and storage conditions), cell lines, and reagents and will likely vary between laboratories. Therefore, a negative culture result might not exclude the presence of infectious virus. Furthermore, the relationship between infectivity in laboratory cell lines in vitro and infectivity in the upper respiratory tract of humans in vivo is poorly defined.
During infection, viral load increases rapidly to a peak and then decreases more slowly. It was not possible for us to know if the Ct values were increasing or decreasing, however, as the usable datasets for the purposes of our analysis had single test results only, and test-and-trace data cannot identify sources of infection. Our analysis assumes that the same viral load produces the same risk of infectiousness and the same risk of false negative LFT results. These relationships are likely to be highly complex. Given that infected individuals have a discreet period of infectiousness, the risk of secondary transmission in those identified earlier, when viral load is increasing, will probably be higher than in those identified later when viral load is decreasing, as these individuals will remain infectious for longer.1 Moreover, the relative infectiousness of an individual with a given Ct value in the viral growth phase may be higher than the same Ct value in the viral decline phase. Potential changes in the ratio between quantity of RNA and live culturable virus during infection are not well characterised because of a lack of longitudinal sampling from recently infected individuals with concurrent quantification of viral RNA copy number and viral culture. For example, host immune response mediators present in samples taken later in infection might impair the virus’s ability to infect cells. These relationships could also change with different SARS-CoV-2 variants and vaccination status, to an unknown extent.
Swabbing technique (including differences between self-testing or assisted testing) might affect the quality of the sample and therefore the relationships investigated in our analysis. We chose, however, to take a pragmatic approach, as this reflects how the testing system currently operates in the real world.
Difficulties also exist in standardising Ct values between laboratories, which can be affected by, for example, sample volumes, the assay used, and the gene target. Some laboratories have performed small calibration experiments to generate equations that link Ct values to viral copy numbers for the assays they use, but important limitations remain in comparability between laboratories and applicability of Ct values to predict in vivo viral load. The use of conversion equations does not account for differences in sample runs, and the amount of viral RNA detected on a swab might not reflect that in vivo (as a result of sampling quality).45
Ct values used in the datasets that formed this analysis were only measured at a single point in time. We acknowledge that some testing procedures are not that simple, with strategies involving serial testing being considered. Given that in the UK the standard is testing at most twice a week, we feel our analysis is representative. Daily LFT testing might, however, mitigate the consequences of low sensitivity and warrants further investigation to generate empirical data.
The findings in this analysis therefore must be taken as illustrative and not exact (providing indicators of cases that might be missed), given these concerns around the validity of the data collated, the use of Ct values as a common metric to connect detection with outcomes, and the approximations we have made in extracting these data. These data are, however, currently the best available and clearly show that missing people with current infection or who are infectious is possible in all settings and likely when applied to community based public health interventions, and the impact of the low sensitivity of LFTs differs considerably between different settings and applications. Even for the same application, the impact may change depending on whether the epidemic curve is rising or falling.46
Conclusions
Allowing for the uncertainties in the results from our analyses, the proportion of people with current infection missed by the Innova LFT is likely to be of public health importance, particularly in settings with greater proportions of infectious people with lower viral loads; where the tests are often being applied. Claims that Innova detects the “vast majority of infectious cases” are not supported. Key models have failed to appropriately use empirical evidence to inform assumptions of test accuracy and chances of infectiousness, resulting in unrealistic overestimates of test performance. Although the public health benefits of using LFTs within asymptomatic screening programmes might outweigh these concerns about sensitivity, and be preferred to not performing any asymptomatic testing, clinicians and policy makers must be aware of the limitations and formal cost effectiveness analyses should be performed. Until new generation LFTs are available that meet the regulatory performance requirements,27 negative test results from LFTs cannot be relied on to exclude current infection. Policy makers need to ensure that the public are aware of the risk of being infectious despite testing negative, and that tests are not used in situations where the consequences of false negative results are considerable.
What is already known on this topic
Lateral flow tests (LFTs) for SARS-CoV-2 are recommended for widespread use, particularly to detect infection in those without symptoms
LFTs only detect SARS-CoV-2 when large quantities of virus are present in samples—individuals with high levels are more likely to have active infection and be infectious than those with lower levels; hence the tests have been advocated for detecting infectiousness
No studies have directly evaluated the proportion of infectious people that LFTs detect because no reference standard exists for infectiousness, and current policy has been supported by predictions from mathematical models rather than empirical evaluations
What this study adds
The proportion of people with markers of infectiousness missed by the Innova LFT depends on the distribution of viral loads in the population tested
When testing people without symptoms, higher proportions of people who are infectious are most likely to be missed
The current analysis of these empirical data highlights inaccuracies in the assumptions made by key models about the relationships between viral loads and infectiousness, resulting in an overestimation of test sensitivity
Ethics statements
Ethical approval
Not required.
Data availability statement
See supplementary file for data used in this analysis.
Acknowledgments
We thank David Jackson, Jade Cogdale, and colleagues at Public Health England for the use of the Ct to RNA copy number conversion equation.
Footnotes
Contributors: JJD conceived the study and wrote the first draft of the paper. JJD and AJS undertook data extraction and analysis and interpreted findings. AS, HH, SH, JD, and AL contributed detail on virological, infectious disease and public health aspects of the study, and contributed to interpretation of the findings. All authors reviewed, commented on, and approved the manuscript. JJD is the guarantor and accepts full responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
Funding: JJD and AJS are supported by the National Institute for Health Research (NIHR) Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham. AS, HH, SG, and AL are supported by the NIHR Health Protection Research Unit in Respiratory Infections, National Heart and Lung Institute, Imperial College London. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: JJD is member of the Medicines and Healthcare Products Regulatory Agency’s In-Vitro Diagnostics external advisory group and contributed to the Birmingham University evaluation of Innova. He has grant funding from The Foundation for Innovative New Diagnostics for the Cochrane review evaluating the accuracy of tests for SARS-CoV-2. AJS reports National Institute for Health Research /UK Research and Innovation funds separate to this study. no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
The lead author (JJD) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
Dissemination to participants and related patient and public communities: No study participants existed to whom we could disseminate the results of this study. We will disseminate the findings from this work across through our university web pages (using plain language summaries), via social media, and through engaging further with media, as appropriate.
Provenance and peer review: Not commissioned; externally peer reviewed.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.