The "false positive paradox" and risks of testing asymptomatics
It is becoming more widely known that lateral flow (antigen) tests and also PCR tests are far less accurate than previously thought. Recent warnings, such as one issued recently by FDA about a specific lateral flow (antigen) test, are just one part of a far broader problem related to screening tests and related policy.
Widespread screening during previous outbreaks and pandemics has generally not been recommended because of the potential for high false positives. The Center for Disease Control (CDC)'s 2004 guidance from the SARS pandemic (CDC 2004), for example, stated: “To decrease the possibility of a false-positive result, testing should be limited to patients with a high index of suspicion for having SARS-CoV disease.”
The World Health Organization (WHO) and CDC did, however, recommend testing of asymptomatic people early in the COVID-19 pandemic, but the CDC revised this guidance in August of 2020 to recommend not testing asymptomatics even after potential exposure, only to reverse course again after public and expert pushback in the U.S. (Fang, M. 2020).
The US Food and Drug Administration (FDA) issued a strongly worded letter to healthcare providers in November 2020 warning about the potential for false positives from antigen testing, describing the problems associated with screening populations with a low background prevalence of COVID-19 (FDA 2020). The letter reminds practitioners that: “As disease prevalence decreases, the percent of test results that are false positives increase.”
CDC’s most recent (March 2021) guidance does, however, still recommend widespread screening, which necessarily includes testing of mostly asymptomatics, despite the widely known issues regarding such policies. CDC’s guidance states: “Rapid, point-of care serial screening can identify asymptomatic cases and help interrupt SARS-CoV-2 transmission. This is especially important when community risk or transmission levels are substantial or high.”
Many countries, including the UK and the U.S., have engaged in widespread population testing (Mercer and Salit 2021: COVID-19 testing is the “largest global testing programme in history, in which hundreds of millions of individuals have been tested to date.”).
Even a test with a very high 99% specificity (1% chance of false positives), when used to screen asymptomatic populations with a low background rate of actual infection, will yield high levels of false positives.
The background rate of COVID-19 infection, even during high points, has always been relatively low. For example, Sadoff et al. 2021, the published results of the Johnson & Johnson vaccine clinical trial, including almost 40,000 participants in half a dozen countries, from late September 2020 to late January 2021, found a 0.5% PCR positive baseline (see Sadoff et al. 2021 supplementary appendix, p. 23).
Similarly, Baden et al., 2020, found a 0.6% background positive PCR test result in the 30,420 clinical trial participants for the Moderna vaccine, after initial testing. Study participants for this trial were selected based on being at higher risk for exposure to the virus and the testing was conducted from late July to late October 2020.
Common sense would suggest that a test with 99% specificity would return only about 1 in a 100 false positive results. But this is not how it works. The false positive rate is far higher when disease prevalence is as low as the studies just cited have found. In other words: the Positive Predictive Value of screening testing is very low when background prevalence is low (Bokhorst et al. 2012; Skittrall et al. 2020; Dinnes et al. 2021).
Here’s why: If we test 1,000 people randomly in a population where 1% have the illness at issue, and our test is 99% specific to that illness, we will have one true positive and one false positive for each 100 tests. So testing 1,000 people results in 10 true positives and 10 false positives.
Using the BMJ test accuracy calculator (Watson 2020; see link for the calculator in the references; it’s educational to play with the calculator to see how different inputs affect false results)., we calculated various scenarios using real-world background prevalence data and test accuracy data.
First, we conservatively assumed 1% pre-test probability of active infection, which is, based on the data reviewed above, which is a higher level of active infection than was found in the large vaccine clinical trials.
We also assumed 58% sensitivity and 99% specificity, which are the findings of a recent Cochrane meta-analysis combining 64 published studies of antigen test accuracy, when used to test asymptomatics (Dinnes, J. et al. 2021).
The result in this scenario is 50% false positives (1 true positive and 1 false positive) — even with a 99% specificity test. There would theoretically be zero false negatives, so the risk of missing actual infections is not at issue.
50% is the same as random chance. In other words, this 99% specificity test can do no better than a coin flip when declaring a positive result. So screening in this scenario is not warranted because data that is no better than a coin flip is not data — it’s random chance.
However, the situation is much worse than this because neither PCR nor antigen tests are close to a 99% specificity level in practice, for various reasons (Braunstein et al. 2021). Lee 2020 performed a lab analysis of the CDC PCR test accuracy, which was widely used in the first months of the pandemic, and found it had a 70% specificity (i.e. 30% false positives) and 80% sensitivity (20% false negatives). This is because of faulty designs built in to the test from the beginning, as various news accounts from the Washington Post, NPR and ProPublica have since revealed.
This level of inaccuracy matches the CDC’s own internal report that found 33% false results when its PCR test was released in late February 2020, as reported on by National Public Radio (Temple-Raston 2020).
Intuitively, and in an emergency situation, we may think that a 70-80% accuracy rate is far from perfect but may still be “good enough.” But this is where common sense and intuition gets us – and the public – into trouble. If we input these figures in the BMJ calculator, we obtain a catastrophic 30 out of 31 false positives.
In other words, at a 1% pre-test probably (background prevalence), just one out of 31 positive test results is a true positive. And, again, we have zero false negatives, so the tests are not missing true positives in this scenario.
This problem relates to more than just misidentifying positive COVID-19 cases; it also is relevant to data on hospitalizations and death rates. After testing became widely available, it became standard practice to test all patients admitted to hospitals in the U.S., regardless of symptoms. While this may have been a necessarily cautious step in order to minimize outbreaks in hospitals, it significantly inflated hospitalizations and deaths attributed to COVID-19. A positive test result was the primary basis for defining COVID-19 hospitalizations and deaths since no symptoms were required to designate a COVID-19 hospitalization or death as such.
In other words, since the CDC and WHO case definitions took the unprecedented step of defining a “confirmed case” as simply a positive lab test result, and then most jurisdictions also defined a COVID-19 hospitalization and a COVID-19 death in the same manner, if the large majority of positive test results are false positives, it is necessary to re-examine the pandemic surveillance data chain from the beginning.
1. Baden LR, El Sahly HM, Essink B, Kotloff K, Frey S, Novak R, et al. Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine. N Engl J Med. 2021;384(5):403–16.
2. Bandler J et al. “Inside the fall of the CDC.” ProPublica. 2020 Oct 15.
3. Bokhorst LP, Zhu X, Bul M, Bangma CH, Schröder FH, Roobol MJ. Positive predictive value of prostate biopsy indicated by prostate-specific-antigen-based prostate cancer screening: trends over time in a European randomized trial*. BJU International. 2012;110(11):1654–60.
4. Braunstein GD, Schwartz L, Hymel P, Fielding J. False Positive Results With SARS-CoV-2 RT-PCR Tests and How to Evaluate a RT-PCR-Positive Test for the Possibility of a False Positive Result. Journal of Occupational & Environmental Medicine. 2021;63(3).
5. CDC 2004, Supplement F: Laboratory Guidance - CDC [Internet]. CDC Notice regarding 2003 SARS. [cited 2021Jun15]. Available from: https://www.cdc.gov/sars/guidance/f-lab/downloads/f-lab-full.pdf
6. CDC Coronavirus Disease 2019 (COVID-19)2020 Interim Case Definition, Approved April 5, 2020 [Internet]. CDC Interim Case Definition. CDC; 2020 [cited 2021Jun15]. Available from: https://wwwn.cdc.gov/nndss/conditions/coronavirus-disease-2019-covid-19/...
7. CDC Prostate Cancer website. What Are the Benefits and Harms of Screening for Prostate Cancer? [Internet]. CDC Prostate Cancer Screening. CDC; 2020 [cited 2021Jun15]. Available from: https://www.cdc.gov/cancer/prostate/basic_info/benefits-harms.htm.
8. Connors E, Williams C. Coronavirus (COVID-19) Infection Survey pilot: 2 July 2020 [Internet]. Coronavirus (COVID-19) Infection Survey pilot - Office for National Statistics. Office for National Statistics; 2020 [cited 2021Jun15]. Available from: https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/...
9. Dinnes_J, Deeks_JJ, Berhane_S, Taylor_M, Adriano_A, Davenport_C, Dittrich_S, Emperador_D, Takwoingi_Y, Cunningham_J, Beese_S, Domen_J, Dretzke_J, Ferrante di Ru(ano_L, Harris_IM, Price_MJ, Taylor-Phillips_S, Hoo-_L, Leeflang_MMG, McInnes_MDF, Spijker_R, Van den Bruel_A, Cochrane COVID-19 Diagnostic Test Accuracy Group. Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection. Cochrane Database of Systematic Reviews 2021, Issue 3. Art. No.: CD013705.
10. Fang M. CDC Quietly Changes Testing Guidelines To Exclude People With No Symptoms [Internet]. HuffPost. HuffPost; 2020 [cited 2021Jun15]. Available from: https://www.huffpost.com/entry/cdc-testing-guidelines-coronavirus-no-sym...
11. FDA 2020. Potential for False Positive Results with SARS-CoV-2 Antigen Tests [Internet]. U.S. Food and Drug Administration. FDA; [cited 2021Jun15]. Available from: https://www.fda.gov/medical-devices/letters-health-care-providers/potent...
12. Flender S. The false positive paradox [Internet]. Medium. Towards Data Science; 2019 [cited 2021Jun15]. Available from: https://towardsdatascience.com/the-false-positive-paradox-f86448a524bc
13. Lee, S. Testing for SARS-CoV-2 in cellular components by routine nested RT-PCR followed by DNA sequencing. International Journal of Geriatrics and Rehabilitation 2(1):69- 96, July 17, 2020.
14. Madrigal A, Meyer R. Why Trump's Rapid-Testing Plan Worries Scientists [Internet]. The Atlantic. Atlantic Media Company; 2020 [cited 2021Jun21]. Available from: https://www.theatlantic.com/health/archive/2020/10/do-rapid-antigen-test...
15. Mercer TR, Salit M. Testing at scale during the COVID-19 pandemic. Nature Reviews Genetics. 2021 May 4.
16. Sadoff J, Gray G, Vandebosch A, Cárdenas V, Shukarev G, Grinsztejn B, Goepfert PA, Truyers C, Fennema H, Spiessens B, Offergeld K, Scheper G, Taylor KL, Robb ML, Treanor J, Barouch DH, Stoddard J, Ryser MF, Marovich MA, Neuzil KM, Corey L, Cauwenberghs N, Tanner T, Hardt K, Ruiz-Guiñazú J, Le Gars M, Schuitemaker H, Van Hoof J, Struyf F, Douoguih M; ENSEMBLE Study Group. Safety and Efficacy of Single-Dose Ad26.COV2.S Vaccine against Covid-19. N Engl J Med. 2021 Jun 10;384(23):2187-2201. doi: 10.1056/NEJMoa2101544. Epub 2021 Apr 21. PMID: 33882225.
17. Skittrall JP, Fortune MD, Jalal H, Zhang H, Enoch DA, Brown NM, et al. Diagnostic tool or screening programme? Asymptomatic testing for SARS-CoV-2 needs clear goals and protocols. The Lancet Regional Health - Europe. 2021;1:100002.
18. Temple-Raston D. CDC Report: Officials Knew Coronavirus Test Was Flawed But Released It Anyway [Internet]. NPR. NPR; 2020 [cited 2021Jun15]. Available from: https://www.npr.org/2020/11/06/929078678/cdc-report-officials-knew-coron...
19. Voysey M, Clemens SAC, Madhi SA, Weckx LY, Folegatti PM, Aley PK, Angus B, Baillie VL, Barnabas SL, Bhorat QE, Bibi S, Briner C, Cicconi P, Collins AM, Colin-Jones R, Cutland CL, Darton TC, Dheda K, Duncan CJA, Emary KRW, Ewer KJ, Fairlie L, Faust SN, Feng S, Ferreira DM, Finn A, Goodman AL, Green CM, Green CA, Heath PT, Hill C, Hill H, Hirsch I, Hodgson SHC, Izu A, Jackson S, Jenkin D, Joe CCD, Kerridge S, Koen A, Kwatra G, Lazarus R, Lawrie AM, Lelliott A, Libri V, Lillie PJ, Mallory R, Mendes AVA, Milan EP, Minassian AM, McGregor A, Morrison H, Mujadidi YF, Nana A, O'Reilly PJ, Padayachee SD, Pittella A, Plested E, Pollock KM, Ramasamy MN, Rhead S, Schwarzbold AV, Singh N, Smith A, Song R, Snape MD, Sprinz E, Sutherland RK, Tarrant R, Thomson EC, Török ME, Toshner M, Turner DPJ, Vekemans J, Villafana TL, Watson MEE, Williams CJ, Douglas AD, Hill AVS, Lambe T, Gilbert SC, Pollard AJ; Oxford COVID Vaccine Trial Group. Safety and efficacy of the ChAdOx1 nCoV-19 vaccine (AZD1222) against SARS-CoV-2: an interim analysis of four randomised controlled trials in Brazil, South Africa, and the UK. Lancet. 2021 Jan 9;397(10269):99-111. doi: 10.1016/S0140-6736(20)32661-1. Epub 2020 Dec 8. Erratum in: Lancet. 2021 Jan 9;397(10269):98. PMID: 33306989; PMCID: PMC7723445.
20. Watson J, Whiting PF, Brush JE. Interpreting a covid-19 test result. BMJ. 2020;:m1808. Online at https://www.bmj.com/content/369/bmj.m1808/infographic.
Competing interests: No competing interests