Why covid-19 antibody tests are not the game changer the UK government claims

The meaning of accuracy

The Cochrane review, an advance copy of which The BMJ has seen, concludes that the data supporting existing antibody tests are so vague that it’s impossible to know how accurate the tests are, especially for people with mild or no symptoms or after symptoms have gone.

Test accuracy, says Matt Keeling, professor of populations and disease at the University of Warwick, is measured in sensitivity and specificity. High sensitivity means that if you’ve previously been infected with SARS-CoV-2 the test will correctly identify this, while high specificity means that if you haven’t been infected the test will correctly identify that. “This is important, as any [people with] false positives could incorrectly assume they have had the virus and therefore are a lower risk,” he explains.

At the time of writing two of the world’s leading suppliers of covid-19 antibody tests are the drug companies Abbott and Roche. In May the UK government purchased 10 million test kits from the two companies after validation from Public Health England (PHE) that they worked. Media reports at the time stated that the Abbott and Roche tests boasted 99% and 100% specificity, respectively, but these figures came from the companies themselves and were based on those initial PHE studies and their own marketing material rather than peer reviewed research.

Roche’s marketing material claims a sensitivity of 100% 14 days after a patient was confirmed to have covid-19 through a polymerase chain reaction (PCR) test.2 Abbott claims 100% accuracy 17 days after symptom onset.3 The PHE studies evaluating the kits use date of symptom onset.4 The discrepancy between the times used makes it difficult to compare the two test kits, yet alone standardise them for use by the NHS.

The accuracy of a test relates to whether it makes errors: whether there are people with the disease who wrongly get negative test results, and whether there are people without the disease who wrongly get positive test results. “Saying a test is 100% accurate implies to the public that neither of these two types of error occurs,” says Deeks. “The reports from PHE make it clear that this statement is misleading.”

A spokesperson for Roche Diagnostics UK told The BMJ, “Our test is designed to be used as an aid to identify who has previously been exposed to the virus and has been through a rigorous regulated process . . . This is based on extensive testing and validation, including the measurement of over 5400 samples.

“We are rolling out antibody tests to the NHS as part of the crucial next step in understanding the spread of this virus, and providing greater confidence and reassurance as we move into the next phase of our response to this pandemic.”

Testing the testers

However, it’s not just the tests that are uncertain. Even the studies evaluating the tests have problems, as the Cochrane review found. Standards for reporting diagnostic accuracy studies have been around for a decade,5 says Deeks, yet none of the 54 studies the Cochrane review saw fit those. “We cannot properly tell anything about the research until the researchers start following those guidelines.”

For instance, PHE’s report on Roche’s test has changed since the original study was posted online. The controversial claim to 100% accuracy was based on a subgroup of patients who had the longest time between the start of symptoms and being tested for antibodies and had originally consisted of just eight patients tested after six to seven weeks. But the report The BMJ accessed online at the time of writing includes patients without symptoms, boosting the number of patients to 10 and reducing the accuracy to a score of nine out of 10.6 A PHE spokesperson told The BMJ that this update was due to two samples being incorrectly labelled as missing interval data and that this had been noted in the document control section of the report.

More importantly, the assessments of the tests were based on samples and not patients—and almost certainly some patients will have contributed multiple samples that will make the results look more precise than they actually are, says Deeks.

“Where will these tests be used? That’s where you do the study,” he says. “The samples [used in evaluating these tests] are probably from the patients in hospitals, who will be the most severely affected, and their antibody response is not that of people with mild symptoms or those who are asymptomatic.”

Because the origin and severity of disease in the samples are not known, it is not possible to check whether they are representative of the typical patient groups receiving an antibody test in real life. And patients without covid-19 but with similar respiratory illnesses were not included, so it’s difficult to tell whether false positives might arise from such people. Moreover, as the studies were undertaken in expert PHE laboratories, the performance of the tests when used in practice may not be as good.

According to Sheila Bird from Edinburgh University’s College of Medicine and Veterinary Medicine there are several problems with PHE’s evaluation of the Roche and Abbott tests, including quality of samples and the absence of data on age and sex, sample sizes, and use of repeat samples.

PHE acknowledged that a small number of samples were repeat samples from different time points in the patient’s disease. It said that samples reflected the general population. “Our evaluations have been completed in record time using the samples and tests that were available to us. We are confident that the volume of samples and methodology was of a high standard,” its spokesperson said.

“Any laboratory using these tests is still required to complete their own evaluation to ensure the tests perform as described—our work is designed to reduce the amount of local work required. Our evaluation work is ongoing and, as more tests become available, we will continue to refine our approach.”

Purpose unclear

The big question is: what will these tests actually be used for?

“What people really want to know from these tests is: am I safe from infection?” says Al Edwards, associate professor in biomedical technology at Reading University. “These tests, at the moment, can’t answer that.”

Neither the Abbott nor the Roche test detects antibodies against SARS-CoV-2’s outer “spike protein,” which studies have indicated are most important for neutralising the virus. Both tests detect antibodies for a different protein termed N.

Moreover, although antibody tests can measure the concentration of antibody in the blood, we don’t yet know what level is related to protection. The measles vaccine offers lifetime immunity, but the most effective cholera vaccine, Dukoral, provides antibodies that protect for only five years.

In fact, there’s currently no evidence that covid-19 antibodies confer immunity to the virus at all—and it’s conceivable they won’t, says Edwards. Immunity passports are a nice idea but may well never come to fruition.

In a statement to The BMJ the Department of Health and Social Care for England said, “We do not currently know how long an antibody response to the virus lasts, nor whether having antibodies means a person cannot transmit it to others.” But it reiterated that antibody testing “will play an increasingly important role as we move into the next phase of our response to this pandemic.”

It further said, “Antibody testing is helping us learn about the level and length of immunity following infection and how the virus is spreading across the country,” adding that PHE is currently running the SIREN study of 10 000 healthcare workers to establish whether antibodies indicate any kind of immunity to covid-19.

Epidemiological studies such as SIREN could well make use of antibody tests. Wider testing could measure the spread of the disease across the country, identifying vulnerable population groups and geographical regions. The Chinese city of Wuhan, for instance, used antibody tests to check its population of around 10 million as it eased out of lockdown. But the authors of the 24 June letter to The BMJ believe that such use is not in the UK government’s plans.

Will Irving, professor of virology at the University of Nottingham and a signatory to the letter, said using antibody tests for epidemiological surveys was possible but would need the structure of a formal epidemiological study, taking into account geography, ethnicity, age, and sex. “This is not the same as simply testing a random set of volunteers who wish to know their antibody status,” he said.

Wasted money?

So what do we know about the currently available covid-19 antibody tests? They are not a good indicator of current infection or for telling people to isolate, says Andrew Preston, reader in microbial pathogenesis at the University of Bath. They are much more accurate if used 14 days after symptoms arise, but currently people with symptoms are told to self-isolate for 14 days anyway, making taking a test moot.

The UK government has already spent £16m (€18m; $20m) buying antibody tests from China that proved inaccurate, many of which now lie in storage. When it ordered 10 million of the Roche and Abbott tests, financial details of the deal were not disclosed. If the Abbott test is supplied at cost, however, there is evidence to indicate that the NHS will spend £79 per test.7

The demand for an expensive, fast turnaround on uncertain tests—with NHS trusts required to offer tests at short notice, building up to testing thousands of samples a day8—is arguably a waste of public money, given that the tests are neither clinically urgent nor a public health priority.

Irving told The BMJ, “The government has been focusing on arbitrary and meaningless targets, such as numbers of tests per day, without any regard for the clinical value of those tests, whether or not the tests are actually carried out, or whether the results are returned in a timely fashion to someone who knows the patient and is able to interpret those results.”

The government’s capacity target for 200 000 tests by the end of May included antibody testing as well as PCR tests. By 31 May it had carried out 23 000 antibody tests, and as at 2 June testing capacity included 40 000 antibody tests. But, says Irving, “Focusing simply on numbers of tests done, without any consideration of the quality of those tests results, is contrary to the basic principles of pathology testing.”

“I think the government bought the tests because they were caught out in not having the reagents or the tests for virus testing early on,” says Preston. “The idea of buying antibody tests to create immunity passports looked attractive, but using them like that is some way off.

“If they’re not being used for full epidemiological studies, I think their main function will be in deciding who to vaccinate, assuming the vaccine comes. That will be useful—but I’m not sure it’s a game changer.”