Testing for antibodies to SARS-CoV-2BMJ 2020; 371 doi: https://doi.org/10.1136/bmj.m4288 (Published 11 November 2020) Cite this as: BMJ 2020;371:m4288
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
We write in response to the BMJ article by Mulchandani et al (1) and the associated editorial which references an online pre-print of our work (2). Like Mulchandani et al, we note the difficulties in evaluating new assays designed to detect antibodies to SARS-CoV-2 virus due to the lack of a gold standard reference test to evaluate against. Statistically unbiased estimation of test sensitivity (the ability of the test to correctly identify true positives i.e. the rate of false negatives) is at present difficult. However, we caution that a direct comparison between the studies cannot be made, given there are clear differences between usage target and reference standards used.
In our pre-print (2) we report longevity of SARS-CoV-2 antibodies for over 20 weeks from time of symptoms or PCR positive status and we describe a laboratory evaluation of the AbC-19 test. In our evaluation of AbC-19 the target condition was antibodies – specifically SARS-CoV-2 Spike protein IgG antibodies. We evaluated the performance characteristics following guidance in the MHRA (3) target product profile which stipulates use of at least 200 samples confirmed as positive and at least 200 samples confirmed as negative. Samples for cohort inclusion were characterised by three available SARS-CoV-2 laboratory-based immunoassays (Roche Elecsys, Abbott Architect SARS-CoV-2 IgG and EuroImmun IgG ELISA) to determine samples as antibody positive or antibody negative. To create a positive cohort we do apply a reference standard which required a positive result in two of three assays; Euroimmun ELISA and one other of Roche or Abbott.
We note Mulchandani et al uses PCR positive status or Roche positivity, whilst they also model two immunoassays together (Roche and Euroimmun ELISA) as a composite reference standard. Roche detects IgG, IgM & IgA antibodies to the nucleocapsid portion of the virus. We prioritise Euroimmun ELISA (IgG to spike protein) alongside either Abbott or Roche positivity to evaluate AbC-19 which detects IgG antibodies to the full trimeric spike protein of SARS-CoV-2; the exact immunogenic protein of the SARS-Cov-2 virus that is used in global vaccines under development. The AbC-19 test could be used for assessing antibody responses to vaccination programmes in our battle against COVID-19.
We acknowledge statistically unbiased estimation of test sensitivity is difficult without an accurate and applicable single gold standard. We use antibody positive as an alternative to RT-PCR positive status as a standard for assessing SARS-CoV-2 antibody assays as our target condition is SARS-CoV-2 IgG antibodies. False positive PCR results may occur in the UK at a rate between 0.8- 4.0% (4) and false negative rates of RT-PCR in up to 30% (5). Furthermore, lack of detectable IgG (up to 90 days post symptom onset) was demonstrated in approximately one in 16 individuals despite previous RT-PCR confirmed infection (6).
The details of our analysis were originally described in a pre-print article (medRxiv) describing the performance of this test to detect antibodies across 654 samples. The cohorts were originally characterised as-
• Negative for SARS-CoV-2 IgG antibody- 350 negative by all three immunoassays- Euroimmun ELISA, Roche and Abbott.
• Positive for SARS-CoV-2 IgG antibody- 304 positive for IgG antibody to spike protein by Euroimmun ELISA and one other of Roche or Abbott.
The performance metrics of AbC-19 lateral flow test was reported against these characterised samples in a laboratory setting as 100% specificity and 97.7% sensitivity.
In the most recently updated medRxiv pre-print article (2) we extended the cohort and tested additional samples to include 818 samples with positive and negative cohorts characterised as-
• (1a) Negative Cohort- 223 pre-pandemic samples were assumed negative regardless of how they were characterised by all three immunoassays- Euroimmun ELISA, Roche and Abbott.
• (1b) Negative Cohort- 265 known negatives for SARS-CoV-2 IgG antibody- were negative by all three immunoassays-Euroimmun ELISA, Roche and Abbott.
• (2) Positive for SARS-CoV-2 IgG antibody- 330 were positive for IgG antibody to SARS-CoV-2 spike protein by Euroimmun ELISA and one other of Roche or Abbott.
The performance metrics of AbC-19 for the detection of antibodies in negative cohort 1a+1b (n=488) in a laboratory setting is 99.59% specificity and 97.58% sensitivity. These metrics were calculated without extrapolating PPV’s to a set prevalence given the number of assumptions that would be required to be implemented and justified.
Using only the pre-pandemic negative cohort 1a (n=223), a robust approach assuming negative antibody status regardless of characterisation on any other immunoassays used for the reference standard, and addressing the concern of bias, had minimum impact on performance metrics showing a 99.55% specificity.
It is important to stress we do not assess AbC-19 for its utility in detecting previous infection, our target condition is presence of antibodies. Our study also assessed analytical specificity of AbC-19 reviewing 34 samples from other known respiratory viruses confirming no cross-reactivity. We highlight faint T lines may be reflective of the longer time from infection for some of the Northern Ireland cohort used (up to 20 weeks post infection) and if AbC-19 is to be used in clinical settings it is important to determine if all users observe the same results as observed in this laboratory evaluation. We highlight high-throughput laboratory evaluation of thousands of samples, even when performed in batches, may result in sample drying on the AbC-19 test leading to incorrect read outs. Additional studies in different use cases with different users are needed depending on application.
We welcome further clinical evaluation of AbC-19 in large cohorts of symptomatic and asymptomatic individuals alongside larger longitudinal studies assessing COVID-19 outcomes in individuals to fully validate its implementation across all intended use cases.
Authors have not been paid or financially benefitted from this study. Advisory CIGA Healthcare roles were unpaid temporary roles. The authors clarify consumable only costs (assays and laboratory expenses) AbC-19 laboratory evaluation are to be claimed from UK-RTC as is normal practice. The manuscript and associated data have only been used to build confidence into the overall device design and performance assessment of AbC-19 and such work was never commissioned for any government contractual consideration.
1. Mulchandani R, Jones H, Taylor-Phillips S, et al., EDSAB-HOME and COMPARE Investigators Accuracy of UK Rapid Test Consortium (UK-RTC) “AbC-19 Rapid Test” for detection of previous SARS-CoV-2 infection in key workers: test accuracy study. BMJ2020;371:m4262
2. Robertson et al. SARS-CoV-2 antibody testing in a UK population: detectable IgG for up to 20 weeks post infection. https://www.medrxiv.org/content/10.1101/2020.09.29.20201509v1 doi: https://doi.org/10.1101/2020.09.29.20201509
3. Target Product Profile: antibody tests to help determine if people have recent infection to SARS-CoV-2: Version 2 (Updated 15 October 2020) https://www.gov.uk/government/publications/how-tests-and-testing-kits-fo.... Accessed 12 November 2020.
4. Surkova E, Nikolayevskyy V, Drobniewski F. False-positive COVID-19 results: hidden problems and costs. Lancet Respir Med [Internet]. 2020 Nov 11; Available from: https://doi.org/10.1016/S2213-2600(20)30453-7
5. Watson J, Richter A, Deeks J. Testing for SARS-CoV-2 antibodies. BMJ [Internet]. 2020;370. Available from: https://www.bmj.com/content/370/bmj.m3325
6. Petersen LR, Sami S, Vuong N, Pathela P, Weiss D, Morgenthau BM, et al. Lack of antibodies to SARS-CoV-2 in a large cohort of previously infected persons. Clin Infect Dis [Internet]. 2020 Nov 4; Available from: https://doi.org/10.1093/cid/ciaa1685
Competing interests: No competing interests
A high sensitivity and specificity of tests that detect antibodies against SARS-CoV-2 proteins with respect to a sub-group of past patients with Covid-19 is no more than encouraging. This provides very limited evidence for or against the ability of such tests to predict immunity as past infection does not guarantee future immunity. Evidence for this can only be obtained by performing the test on a group of people, following them up and then finding out at different time intervals how many develop Covid-19 with symptoms and a positive PCR and also those who need home care, hospital admission, oxygen, ventilation how many die. This is understood clearly by those who organise RCTs to assess the effect of vaccination for SARS-CoV-2, but strangely not by those who assess diagnostic tests. The task is easier for tests because there is no need for randomisation. The outcome probabilities will vary for people who work in hospitals, their patients, those in care homes and in those who live in different communities with different incidences of Covid-19.
If antibody titres are available then it would be far more informative to plot the distribution of these titres in those who subsequently develop a Covid-19 illness and those who do not. This will allow the probability of developing Covid-19 to be calculated over different the time courses in the study, which should be carried out for as long as possible. If the test only created a dichotomous result (i.e. is positive or negative) then this calculation would be based on outcome sensitivity and specificity. The object is to predict a clinical outcome and not some arbitrary diagnostic criterion and easily acquired but questionably relevant data from the past.
Unfortunately those who assess diagnostic tests seem to be focused too much on arbitrary ‘diagnoses’ and not observable or measurable outcomes, which is what clinicians need. They also place too much emphasis on sensitivity and specificity which require dichotomisation of numerical results, thus throwing away information. The Covid-19 crisis has revealed many flaws in the way that countries and health services are run. One of these flaws is the current way in which we assess diagnostic tests. Those who assess tests might get a slightly better insight from reading the Oxford Handbook of Clinical Diagnosis, especially the section on ‘Reasoning with numerical test results’ .
1. Llewelyn H, Ang AH, Lewis K, Abdullah A. Reasoning with numerical test results. In: The Oxford Handbook of Clinical Diagnosis. 3rd ed. Oxford University Press, 2014: 630-1. (See the open access Chapter 13: http://oxfordmedicine.com/view/10.1093/med/9780199679867.001.0001/med-97...
Competing interests: No competing interests