Testing times for the government’s favoured antibody kit

Fast-track approval

The AbC-19 test did not have to wait for independent evaluation before getting a CE mark for professional use in July. Instead, the rules allow the manufacturers to self-assess the test using figures posted on Abingdon Health’s website. The Medicines and Healthcare Products Regulatory Agency (MHRA) recommends that test manufacturers reach a minimum standard of 98% sensitivity (ability to correctly identify a true positive sample) and specificity (ability to correctly identify a true negative sample). Abingdon Health reported that its test had a sensitivity of 98% and specificity of 100%, so was within recommended standards.

The only published evidence on the accuracy of AbC-19, before the publication of The BMJ paper last week, was a preprint article, not yet peer reviewed, reporting results from a study led at Ulster University. The study was funded by members of the consortium and led by professors Jim McLaughlin and Tara Moore, who were recruited as consultants in April by CIGA Healthcare, a member of the UK Rapid Testing Consortium. They found the test to have 97.7% sensitivity and 100% specificity. The study’s methodology has been criticised, including by academics brought together by the Science Media Centre, because it screened out blood samples that were difficult to classify as real positives or not,.5 This approach will tend to overestimate test accuracy.

By contrast, the large scale PHE study, using 4842 blood samples, published in The BMJ, found the test had clinical sensitivity of 92.5% and specificity 97.9% if using known positive and negative samples, with sensitivity falling to 84.7% among cases with unknown infection status, as would be the case in real life settings.

Sylvia Richardson, the Royal Statistical Society’s president elect and chair of its covid-19 taskforce, says that it is vital to get real world estimates of how a test might perform, as PHE had done. “At a time when reliable, effective testing is vital to our ability to contain the covid-19 pandemic, and when the government is proposing to spend many millions on novel tests, it is hugely important that independent assessment of the context-relevant performance of antibody or antigen tests is available prior to purchasing,” Richardson says.

In October, the Department of Health and Social Care announced the purchase of one million AbC-19 tests from the UK Rapid Testing Consortium to support national surveillance studies. In the press release, dated 6 October, health minister Lord Bethell is said to be, “thrilled by the product, both for Britain and export markets around the world.”

Emails seen by The BMJ show that officials at the department knew about the disappointing results of the PHE study before the announcement. The emails also show how the department blocked publication of a preprint of the PHE study.

The plan, according to a Department of Health and Social Care email dated 25 September, was for “minimal mention” of the PHE study in Lord Bethell’s announcement, “but we do need to mention it as we will get asked.”

PHE staff warned that there were “significant risks” in not publishing the PHE evaluation showing the low accuracy of the tests and asked if holding back the results had been agreed by ministers. In an email on 1 October, PHE asked whether this strategy—of holding back results—had been agreed. “Is everyone aligned on the handling—ministers, spads, etc?” it asked. The department replied, “Yes everyone is aligned as far as I know. No. 10 [Downing Street, the prime minister’s office] now aligned.”

Asked whether he was aware that the publication of the PHE preprint had been blocked, Lord Bethell did not reply before The BMJ went to press.

When questioned by The BMJ, the Department of Health and Social Care denied that it had blocked publication and said that it had planned to submit the PHE study to The BMJ for peer review with a view to publication. It added that it had bought the AbC-19 test for surveillance studies only. “They were never intended for, and have never been issued for, widespread public use. This robust evaluation was carried out by PHE at the Department’s request before any purchase was made, and PHE approved the test for use in surveillance studies.”

The PHE study says that the AbC-19 test might be accurate enough for serosurveillance to gauge the level of infection in the population, but only if the test’s accuracy improved, the relation between the test and future immunity was better understood, and the difficulty non-professionals might have accurately reading the test was accounted for. In a press release issued by the Science Media Centre on 7 October, Hayley Jones, senior lecturer in medical statistics at the University of Bristol, said: “When using antibody tests for surveillance purposes (to estimate what proportion of a population has been infected or has antibodies) errors in the accuracy of a test can in principle be corrected for. Test errors could be dangerous, however, if individuals receive their test results and change their behaviour based on this—due to a belief that they are immune from the virus. For now, any individual who receives a positive antibody test result should interpret it with caution and not change their behaviour as a result.”5

Abingdon Health said in a statement to The BMJ: “Our customer, [the Department of Health and Social Care], is satisfied with the performance of the test, as is the UK Rapid Testing Consortium, and will continue to roll out the use of the product.”

The government is currently conducting nine surveillance studies including the UK Biobank SARS-CoV-2 serology study with 20 000 families who self-administer finger prick blood samples once a month, and the REACT 2 study, which measures antibody levels in 150 000 people using finger prick testing. If the AbC-19 test were to be deployed in one of these studies, one in five positive results may be incorrect.

Joshua Moon, research fellow in the Science Policy Research Unit at the University of Sussex, adds: “Although the idea of a pinprick test is very useful in measuring disease spread and assessing public health interventions, overestimating the level of covid-19 in the community by 20% makes you wonder what the use case for this test could be.”

Tender subject

British rapid antibody test manufacturers have been frustrated at the government’s procurement process. Over spring and summer, many of them submitted rapid lateral flow immunoassay tests to the government’s new test approvals group, set up specifically to assess covid-19 diagnostic tests. Every lateral flow test was rejected for failing to meet the MHRA recommended standard, which requires 98% sensitivity and specificity, despite British made tests receiving approval in Europe and the US.

Yet, in early May, the government agreed to spend £13.5m buying Roche’s antibody tests three days before scientists at PHE’s Porton Down laboratory released a study that found the test was 84% sensitive—less accurate than some of the rejected lateral flow tests.

“There is no process, no tender available, no opportunity to bid,” said one manufacturer who wished to remain anonymous. “There are rapid tests from British companies being bought by governments around the world that could have been deployed here six months ago.”

“What is most incomprehensible is the price. Lateral flow is quick and easy to use, and cheap to make—that’s the beauty of it. At these volumes the government should be paying closer to £3 (compared to approximately £10)—especially as they’ve already given [the UK Rapid Testing Consortium] £10m upfront for components and are procuring directly.”

Jolyon Maugham QC, director of Good Law Project, said, “However amazed you are by this bestiary of incompetence, you’re not amazed enough.”

“This [the AbC-19 test contract] was a £75m contract, let without competition, on the basis of profoundly flawed research,” he told The BMJ, “And what we then get is a government that knows of these flaws and tries to suppress their publication.”

Sylvia Richardson of the Royal Statistical Society says that there are lessons to be taken from this case: “More transparency—undertaking robust evaluations more quickly, publishing the results in a timely manner, and acting on them—would help the government avoid costly mistakes about testing.”