Interpreting a covid-19 test result
BMJ 2020; 369 doi: https://doi.org/10.1136/bmj.m1808 (Published 12 May 2020) Cite this as: BMJ 2020;369:m1808Read our latest coverage of the coronavirus pandemic
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Dear Editor
This article adresses a crucial point, with - in my opinion - huge and poorly addressed consequences for the global fight against Covid-19. The (technique of taking a) PCR test isn't as accurate as one would expect from a 'golden standard', with false-negative test results occuring more frequently than many of us know. Indeed, governments make the people believe that a negative PCR carried out by an official health service means they are not infected, and a positive test means they are. But unfortunately, this is not the case, as I will illustrate further.
I am a GP from the Netherlands, unvaccinated. I was on vacation in (my country) with my family, when I developed symptoms of Covid-19 (shortness of breath, cough, fever). I immediately decided to leave the campside and go home, but not after carrying out a self-test (SARS-Ag), which was POSITIVE.
After returning home, I made an appointment at the local health service to confirm my test officially by PCR. To my surprise, this test turned out NEGATIVE! Because I felt this could not be true, I tested myself, the same day, in my practice, with an official PCR, which indeed turned out to be POSITIVE.
Next day, the same thing happened to my wife: slightly symptomatic, first a NEGATIVE PCR by testing at the local health service and subsequent POSITIVE PCR when carrying it out myself. All my three children were tested NEGATIVE on PCR tests by the local health service, but they developed symptoms and positive self-tests in the next 3 days.
Many lessons can be learned from this case.
For example: without the clinician's expertise (taking symptoms and epidemiological data into account), a negative test result for Covid-19 is not far from useless and could contribute to further spreading the virus.
Second: the technique of testing probably influences test results dramatically. All my family members were tested by friendly personel, testing accordingly (i.e. gently). The duration and force with which the swab is being put in the solution, varies highly among testing-personel and can also contribute to false negative test results.
I tried to ring alarm bells on local and national level, unfortunately without any reaction. I hope this response will somehow do.
Competing interests: No competing interests
Dear Editor
The article seems to reflect a 'trend' in the BMJ towards using Bayesian methods in a mixed response to a fast-moving, unpredictable pandemic. The complicated ideas of 'pre-test probability', 'post-test probability', 'subjective beliefs' and 'likelihood ratios' are all Bayesian.
The more slow-moving, lethargic frequentist apporoach is falling behind in the midst of the ugliest of crises..
We need Bayes, Bayes, Bayes, or we just won't be able to quantify and understand the spread of Covid.
One flaw with the piece is no mention in it of predictive values. These take into account the prevalence of a disease. Sensitivity and specificity are theoretical, abstract. They're not so useful to the GP Practice in a local area as negative or positive predictive values. Sensitivity and specificity won't say, if a test result is positive, the chance of actually having Covid. The likelihood ratio necessarily incorporates the underlying disease prevalence, but is less easy to understand than predictive values.
The article also has no ROC (receiver operating characteristic) curve. ROC provides a trade-off between sensitivity and specificity, for different cut-off values.
Sensitivity and specificity, just like type I and type II errors, are in an inverse relationship with each other. So it is no surprise if less sensitivity means more specificity, and vice versa.
Testing has been a nightmare in this pandemic. The article here indicates just how difficult it is to make medical statistics a useful tool. The failure of the authors to refer to predictive values indicates just how confusing their article is.
Meanwhile, back in the hideous real world apart from the sophisticated realm of maths, the self-testing kits being handed out are incomprehensible to the layman such as me.
And we still don't have the easier serology tests.
Oh dear, this pandemic has not been the finest hour for medical statistics. Those five o'clock press conferences were an example of numbers and equations often baffling the mass audience.
The much missed Douglas Altman would have been appalled that the BMJ has given us no educational article on the r-number. No one has done justice to this terribly fascinbating, this unbelievably complex, statistical device.
References:
(1)
Medical Statistics. A Commonsense Approach. 3rd Edition. Michael J. Campbell. David machin. Wiley . 1999.
(2)
How to Read a Paper. Trisha Greenhalgh. BMJ Books.
(3)
Interpretation and Use of Medical Statistics. Fifth Edition. Leslie E. Daly and Geoffrey J. Bourke. Blackwell Science. 2000.
Competing interests: No competing interests
Dear Editor,
The article by Watson et al.[1] provides clear guidance on how to interpret information from manufacturers of laboratory tests on sensitivity and specificity when a certain prevalence is present. However, we are surprised when positive and negative predictive values are calculated from the sensitivity, specificity and prevalence data and what uncertainties can arise even though one feels certain.
We looked particularly at column 3 of table 1. Here we find the probability of getting a false negative result twice in a row and it appears that an error has crept in. If we assume that, for example, there are no interfering substances in the saliva of a patient with a throat swab that could inhibit the PCR examination, the probability of obtaining a false negative result the first time is entered in column 2 of the table. This result should be squared for obtaining a false negative result twice as these are independent probabilities (see Bayes’ Theorem [2]).
Furthermore, since the sensitivity and specificity are doubted in the absence of reference methods (which is quite legitimate and sensible), so unfortunately the prevalence must also be critically questioned. In the absence of a e.g. laboratory /histo-pathological /clinical /radiological reference method, the estimation of prevalence can also be fraught with errors, e.g. asymptomatic carriers, patients with unspecific clinical symptoms and negative tests. Are these cases to be included in the prevalence or not?
Finally, the article was specifically about real-time polymerase chain reaction (rt-PCR) as a detection method. Currently, rapid antigen tests (RATs) are very popular and are considered by many governments to be a recommended procedure for broad, often repeatable mass screening, e.g. for schools, kindergartens, public events. With these tests, the sensitivity and specificity are even lower because, as well illustrated in the article, pre-analytical errors, e.g. in acceptance, are more likely since these are carried out by laypersons. As these tests are widely available, they can and should be repeated, so the interpretation of a repeated false negative test becomes more important in this context.
Jürgen Durner,1,2 *# Miriam E. Draenert,2# Roland Kammergruber, Siegfried Burggraf,1 David C Watts,3 and Marc Becker1,2
1 Laboratory Becker & Colleagues, Führichstr. 70, 81671 Munich, Germany
2 Department of Operative/Restorative Dentistry, Periodontology and Pedodontics, Ludwig-Maximilians-Universität München, Goethestr. 70, 80336 Munich, Germany
3 School of Medical Sciences and Photon Science Institute, University of Manchester, UK.
#Both authors contributed equally to this response
*Corresponding author: Prof. Dr. rer. nat. Dr. med. Jürgen Durner
Department of Conservative Dentistry and Periodontology, University Hospital
Ludwig-Maximilians-University Munich
Goethestraße 70
80336 Munich
Germany
Reference
1. Watson J, Whiting PF, Brush JE. Interpreting a covid-19 test result. BMJ 2020;369:m1808. doi: 10.1136/bmj.m1808 [published Online First: 2020/05/14]
2. Held L, Bové DS. Likelihood and Bayesian Inference. 2 ed. Berlin Heidelberg: Springer 2020.
Competing interests: No competing interests
Dear Editor
The article by Watson et al has a clear explanation and beautiful infographic to show the dependence of predictive values on test accuracy and pre-test probability, and it highlights this with the boxed statement: "Interpreting the result of a test for covid-19 depends on two things: the accuracy of the test, and the pre-test probability or estimated risk of disease before testing".
With SARS-CoV-2 infection, things are a bit more complicated than this statement seems to imply. It is logically correct, but misleading, because it camouflages the clinically important fact that test accuracy varies with symptoms and their duration, viral load, time of inoculation, etc. The discussion around pre-test probability based on the clinical details also misses the opportunity to highlight the dependence of test accuracy on clinical details.
Michael Power
Competing interests: No competing interests
Dear Editor,
I am very pleased about your infographic. It tells the terms of conditional probability in such an easy way.
But it would be even better if it could be used for realistic numbers occuring this days. Maybe you adopt the graphic in the following way:
It schoud work for the commonly used "positives / 100000 Tests" figure. For this I suggest to introduce different symbols for amounts of people like big skyscrapers for 1000 People, small skyscrapers for 100 People, a house for 10 People and the already used circle for a individual.
Also the key numbers of the test should be ernterable as real number up to 3 decimals.
If you don't want to realize this please contact me to find a way of running this project by e-Mail
Andreas Macher
Competing interests: No competing interests
Dear Editor
with respect to my previous Rapid Response [1], I first should like to correct a misprint in the figure for the victims of air pollution in Italy I had quoted from [2], which, for 2016, is 58,600 (not 58,200 as I wrote) from fine particulate matter (PM 2.5). Therefore the estimated overall mortality burden in Italy due to air pollution as caused by three main pollutants (NO 2, O3, PM 2.5) during 2016 according to the European Environment Agency was 76,200.
As regards the relationship between covid-19 and air pollution, in the last EEA report [3, p. 31] we are wisely reminded that "Spatial coincidence alone cannot be taken as causality.'' However chronic exposure to atmospheric pollution is not merely "spatial coincidence'', and it is both a priori and a posteriori clear that it aggravates "co-morbidities that could lead to fatal health outcomes of the virus infection'' [4].
As regards the specificity issue, I wish to add some update and remarks to strengthen the case that a thoughtful and cautious approach to SARS-CoV-2 testing is needed (this applies to a greater or lesser degree to all biomedical testing, of course, cf. "Test Bloater'' in [5]).
The WHO figure for the covid-19-attributed deaths in Italy up to 3 January is 74,985 (that is, less than the 2016 deaths due to air pollution for the mentioned three main pollutants), out of a total figure for confirmed cases of 2,141,201, giving a lethality of 3.5 % (the corresponding calculations for UK and Sweden give, respectively, 2.8 %, and 2.0 %) [6].
Based on these data, the prevalence of confirmed covid-19 in Italy should also be, up to the first decimal figure, 3.5 %. However, this estimate must be compared with the seroprevalence national inquiry, performed from 25 May to 15 July 2020, which estimated a national seroprevalence of 2.5%, although, unsurprisingly, with very big differences among regions (Lombardia – Northern Italy – was at 7.5%, while Campania – Southern Italy – was at 0.7%) [7]. So a 3.0 % prevalence may be closer to the truth.
It has been surmised by some readers that a 95% for specificity is too pessimistic. Now, whatever is guaranteed in laboratory studies, the most optimistic real-world value for specificity appears to have been 0.997 [8]. Under the assumption of a 3.0% prevalence in Italy, testing with sensitivity 0.7 and specificity 0.997 would give a false positive rate of 12.17 %. This means more than 260,000 Italian "cases" might have been false positives.
In my opinion this order of magnitude translates, in terms of restrictions on economic activity and sociality, and of psychological effects on healthy people mistakenly identified (and therefore both perceived and self-perceived ) as infected and infectious, into an iatrogenic national emergency.
Of course the possibility of arguing along these lines depends on the fact that, for the first time in the modern historical record, "cases'' have been allowed by WHO to be based on the outcome of a test alone [8]. As explained in my previous Rapid Response, at the beginning of the pandemic, testing was mostly limited to symptomatic individuals. But as soon as testing kits became more abundantly and widely available, screening and contact tracing got a bigger and bigger place among the causes for testing. To get an idea of the present SARS-CoV-2 testing policy, in Italy, for instance, in the two-week period 19 October-1 November 2020, out of 315,527 "cases'', only 35.1% have been tested because they had symptoms, and in the period 21 December 2020 - 3 January 2021, out of 188,732 "cases'', the corresponding percentage has been 34.2% [9, 10].
A welcome piece of news between these periods is that on last 14 December a "medical product alert'' has been published by WHO on "Nucleic acid testing (NAT) technologies that use real-time polymerase chain reaction (RT-PCR) for detection of SARS-CoV-2" [11], giving the following advice:
"4. Consider any positive result (SARS-CoV-2 detected) or negative results (SARS-CoV-2 not detected) in combination with specimen type, clinical observations, patient history, and epidemiological information.
5. Provide the Ct [cycle threshold] value in the report to the requesting healthcare provider."
I may be not alone in thinking that this alert could have been published a little earlier. It is also doubtful whether it has been readily and fully implemented.
References
[1] M. Mamone Capria: Re: Interpreting a covid-19 test result - Test Specificity is an Important Issue, 23 May 2020, https://www.bmj.com/content/369/bmj.m1808/rr-13
[2] "Air quality in Europe – 2019 report", European Environment Agency, October 16, 2019, www.eea.europa.eu/publications/air-quality-in-europe-2019
[3] "Air quality in Europe — 2020 report'', 23 November 2020, https://www.eea.europa.eu/publications/air-quality-in-europe-2020-report
[4] A. Pozzer et al., "Regional and global contributions of air pollution to risk of death from COVID-19'', Cardiovascular Research, 26 October 2020, https://doi.org/10.1093/cvr/cvaa288
[5] M. Michael III, W.T. Boyce, A.J. Wilcox, Biomedical Bestiary: An Epidemiologic Guide to Flaws and Fallacies in the Medical Literature , Little Brown & Co (T), 1984.
[6] "Weekly epidemiological update - 5 January 2021’’, https://www.who.int/publications/m/item/weekly-epidemiological-update---...
[7] "ISTAT, Primi risultati dell'indagine di sieroprevalenza sul SARS-CoV-2’’, 3 August 2020, https://www.istat.it/it/archivio/246156
[8] A.N. Cohen, B. Kessel, M.G. Milgroom, "Diagnosing COVID-19 infection: the danger of over-reliance on positive test results'', 28 September 2020, https://doi.org/10.1101/2020.04.26.20080911
[9] "Epidemia COVID-19 Aggiornamento nazionale 7 novembre 2020 – ore 11:00,'', 10 November 2020,
https://www.epicentro.iss.it/coronavirus/bollettino/Bollettino-sorveglia...
[10] "Epidemia COVID-19 Aggiornamento nazionale 5 gennaio 2021–ore 12,'' 8 January 2021,
https://www.epicentro.iss.it/coronavirus/bollettino/Bollettino-sorveglia...
[11] "WHO Information Notice for IVD Users’’, 14 December 2020, https://www.who.int/news/item/14-12-2020-who-information-notice-for-ivd-...
Competing interests: No competing interests
Dear Editor,
It would be good if you could change the data entry fields to allow floating point numbers, since the Specificity is actually probably around 99.9% and you can only enter 99%. I guess you'd need to increase the sample size to 10,000 or something as well. The tool could be used better to reflect the current situation then.
It's a very cool tool though, well done!
Competing interests: No competing interests
Dear Editor
After reading this article plus all the rapid responses so far,
plus almost everything published on line in the bmj,
plus newspaper reports-
I have concluded that :
We the public are “ being sold a pup”.
If anyone believes that I have misled myself, I would love to be put wise.
Competing interests: No competing interests
The small miracle of PCR tests
Dear editor,
The small miracle of PCR (polymerase chain reaction) is a very high specificity that allows meaningful testing with few false positives and high positive predictive values even in low prevalence settings. This fortunate fact is unfortunately misrepresented in this practice pointer (1).
I appreciate the goal of this pointer, and its general message that no tests are perfect is valid. All the information about sensitivity and false negatives is important and accurate. However, when it comes to specificity, PCR testing is extremely specific, and this feature of the technology needs to be precisely rendered.
In the practice pointer Watson et al write concerning PCR tests, that "we will use the lower end of current estimates from systematic reviews,6 with the approximate numbers of 70% for sensitivity and 95% for specificity for illustrative purposes". Although the authors write "for illustrative purposes" and also (correctly) state that "a positive covid-19 test result should be very compelling", the choice of 95% specificity is unfortunate.
The "magic" of PCR tests cannot be appreciated without understanding the underlying biology. According to the US National Institutes of Health (NIH):
"Sometimes called 'molecular photocopying,' the polymerase chain reaction (PCR) is a fast and inexpensive technique used to 'amplify' - copy - small segments of DNA. Because significant amounts of a sample of DNA are necessary for molecular and genetic analyses, studies of isolated pieces of DNA are nearly impossible without PCR amplification.
"Often heralded as one of the most important scientific advances in molecular biology, PCR revolutionized the study of DNA to such an extent that its creator, Kary B. Mullis, was awarded the Nobel Prize for Chemistry in 1993." (2).
What has made PCR tests so tremendously useful in this pandemic (odes should probably be sung to their glory), is that they are based on gene technology and identify specific sequences of DNA or, in the case of SARS CoV-2, RNA molecules. Since RNA is extremely specific for a biological entity, this allows extreme specificity and few false positive tests and high positive predictive values - even in contexts with low prevalence.
As another article examining the merits of tests in the context of Covid-19 states, "Specificity of most of the RT-PCR tests is 100% because the primer design is specific to the genome sequence of SARS-CoV-2" (3).
While serology tests are less specific, the same may be true of novel antigen tests. According to the US Centers for Disease Control and Prevention (CDC), "The specificity of rapid antigen tests is generally as high as RT-PCR – the first antigen tests that have received FDA EUAs have reported specificity of 100% – which means that false positive results are unlikely."
https://www.cdc.gov/coronavirus/2019-ncov/lab/resources/antigen-tests-gu...
Misrepresentation of the properties of PCR tests are not uncommon and play into the idea that the current idea that waves of new cases in the Autumn of 2020 are mainly due to false positives (see, e.g. 4)
In a July article in The Spectator, Oxford professor of evidence-based medicine, Carl Heneghan, also refers to the BMJ practice (5). While the pointer used specificity 95% "for illustrative purposes", Heneghan refers to it as "a review" that "reported that the specificity of PCR tests could be as low as 95%". While Heneghan is also only trying to contribute to the understanding of tests, this shows that using a figure like 95% for illustrative purposes may be misunderstood as a reference to the facts, be misleading and give the impression that false positives are very common. Admittedly, Heneghan also writes in The Spectator, that "based on the latest data, the specificity may be as high as 99,9 per cent", but he also states he thinks this is too high.
I do not know just how high the specificity of PCR tests is in clinical practice, and perhaps nobody does. There are many different PCR tests on the market. However, based on the biology underlying PCR tests, the specificity may also be higher than 99,9%. Just what the specificity actually is, is a critically important discussion, which we should have in the major medical journals. While PCR tests possess extreme specificity in the world of tests, nothing is 100% in real life. The JAMA article mentioned above also states that, "Occasional false-positive results may occur due to technical errors and reagent contamination" (3).
In a context where millions of tests are performed in very low prevalence settings, false positives and low positive predictive values may still become a problem. Heneghan has a point here (5). This should be discussed when considering mass screening of very low risk individuals in such settings. But the specificity is not as low as 95%. The discussion on this important issue needs to be more precise.
In the world of covid-19 it is worth stopping to marvel at PCR tests as a tool. Their specificity in making us able to pinpoint just where the virus is at, should be much (although not uncritically) appreciated.
References:
1) Watson et al. https://www.bmj.com/content/369/bmj.m1808
2) NIH (2020): Polymerase Chain Reaction (PCR) Fact Sheet https://www.genome.gov/about-genomics/fact-sheets/Polymerase-Chain-React...
3) Sethuraman et al. 2020. Interpreting Diagnostic Tests for SARS-CoV-2 https://jamanetwork.com/journals/jama/fullarticle/2765837
4) Chief Science Officer for Pfizer Says "Second Wave" Faked on False-Positive COVID Tests, "Pandemic Is Over" https://www.reddit.com/r/LockdownSkepticism/comments/izl5i1/chief_scienc...
5) Heneghan. C. 2020. How many covid diagnoses are false positives? https://www.spectator.co.uk/article/how-many-covid-diagnoses-are-false-p...
Competing interests: No competing interests
Re: Interpreting a covid-19 test result
Dear Editor
Some statements in the article are based on wrong assumptions.
It is assumed that a positive test result is always more informative than a negative test result. This is at least not correct if the pre-test probability is very low.
It is assumed that the test measures on a nominal binary level but it measures on a ordinal level (number of cycles). The result is further converted in a binary result (positive or negative) by choosing a cut-off value. The choice of different cut-off values leads to other sensitivities and specificities. In the article it is supposed that the sensitivity and specificity remains constant and that is only correct for nominal binary measurements.
As in many articles, accuracy is described by sensitivity and specificity but these are in most cases not constant since it is possible to change the cut-off point. The accuracy of the test measuring on ordinal level must then be expressed in a combination of both sensitivity and specificity, in measures such as the odds ratio, the AU(RO)C, Yule’s Q and so on… Also by changing the cut-off point these measures remain constant and reflect the accuracy (validity) of the test. All these measures measure the validity on an ordinal scale except one of them: Yule’s Y. Yule’s Y measures the association on a ratio level and therefore is it the measure of preference.
It should perhaps be recommended that the esults of a study be expressed by mentioning a measure of association, the sensitivity (or specificity), the pre-test probability (prevalence in case-control studies) and the total number of observations so that calculations can be made on all measures that are considered to be important (predictive values, CI’s, significance tests a.s.o.).
Competing interests: No competing interests