Feature Medical Devices

Laboratories and regulator misled over antibiotic susceptibility test discs

BMJ 2013; 346 doi: http://dx.doi.org/10.1136/bmj.f837 (Published 13 February 2013) Cite this as: BMJ 2013;346:f837
  1. Deborah Cohen, investigations editor1,
  2. Glenn Swift, freelance journalist2
  1. 1 BMJ, London WC1H 9JR, UK
  2. 2London, UK
  1. Correspondence to: D Cohen dcohen{at}bmj.com

Patients may have received inappropriate antibiotic treatment as a result of quality problems with one company’s test discs, Deborah Cohen and Glenn Swift report

A leading global producer of diagnostic tests has been falsely marketing one of its products, an investigation by the BMJ shows.

Oxoid, owned by US diagnostics giant Thermo Fisher, has been selling discs for testing antimicrobial susceptibility that do not always contain the advertised amount of antibiotic. The website of the $10bn (£6bn; €7bn) a year company claims that it is “renowned for quality, accuracy, reliability and innovation.” However, internal documents seen by the BMJ show that this reputation may not be entirely justified.

A BMJ investigation shows that the company used flawed data to determine the quality of its products and misled the UK regulator when it was alerted that some discs did not contain antibiotic. The company continued in early 2012 to mislead clinicians by inaccurately labelling some of its antimicrobial susceptibility discs.

At worst, patients may have been prescribed the wrong antibiotic or moved from oral to intravenous treatment unnecessarily. This comes at a time when health agencies around the world are concerned about antibiotic resistance and the shortage of new antimicrobial treatments in the pipeline.

Only last month, the chief medical officer for England, Sally Davies, described antibiotic resistance as an “apocalyptic scenario.”1 “Antibiotics are losing their effectiveness at a rate that is both alarming and irreversible . . . I urge patients and prescribers to think about the drugs they are requesting and dispensing,” she said.

The findings also pose further questions for Europe’s in vitro diagnostics regulators.

Antimicrobial susceptibility test (AST) discs are used in hospitals around the world to guide the choice of antibiotic treatment. The Basingstoke based company—which was originally housed in the Oxo Tower on London’s Southbank—sells discs to countries across Europe and around the world, including, Australia, Canada, India, and Pakistan. They are also sold in the US where they are marketed under the brand name Remel.

Drug companies provide their antibiotic to Oxoid, and the drugs are then used to impregnate small round discs of absorbent paper. In the microbiology laboratory, discs containing specified amounts of antibiotic are put on to an agar culture medium swabbed with a microbial isolate from a patient. When the organism is susceptible to the antibiotic a clear zone forms around the disc. The bigger the zone, the more likely is the antibiotic to be effective against that particular organism.

According to Oxoid’s website this disc diffusion method was developed in 1947 and is the most widely used test of antibiotic susceptibility. “Performed with care, using adequate controls, it is as accurate as more costly and complicated tests,” it says.2

However, emails seen by the BMJ suggest that accuracy may not be Oxoid’s primary concern. Internal documents show that batches of the disc have been released that do not contain the advertised amount of antibiotic and sometimes no drug at all. Sources also told the BMJ that Oxoid does not have any stability data—used to determine the shelf life of a product—an allegation not denied by the company when the BMJ put it to them. Oxoid claims on its website that: “Real-time shelf life studies of the discs in their packaging demonstrate that they meet the stated storage life printed on the labels.”2

Risks of substandard discs

For decades, the published literature has emphasised the importance of correctly stating the amount of drug on the disc: differences in zone sizes of only a few millimetres may alter the interpretation of the test. As Oxoid’s website states, this method helps to identify “resistant, intermediate and susceptible organisms through quantitative results.”

The British In Vitro Diagnostic Association, which represents the diagnostic industry, says: “Using the right antibiotic is fundamental to successful treatment; using the wrong antibiotic wastes time and money, extends the period of illness and may exacerbate the emergence of resistance, endangering other patients.”3

According to David Shlaes, a professor of microbiology and a consultant in AST development in the US, inaccuracy of discs poses risks to patient care. If a disc contains too little of the drug there is a risk that the organism will appear to be resistant and the patient may be moved on to another antibiotic with more side effects or requiring intravenous rather than oral administration, he explains. More seriously, too much antibiotic on the discs risks giving a false impression of susceptibility. This may result in a patient being given an antibiotic that doesn’t work thereby delaying effective treatment.

So how and why are such serious mistakes being made by this successful global brand, which counts the NHS and GlaxoSmithKline among its clients?

Longstanding quality problems

Oxoid clearly understand the need to get the disc content right. “To obtain accurate and reproducible results, a key point is the quality of the discs,” a 2006 company press release states.4

But evidence has emerged that antibiotics were not handled as carefully as they should be—that discs were not always kept in the correct conditions and that there was a risk of cross contamination between discs impregnated with different antibiotics.

In 2006, an internal review of the company’s antibiotic handling procedure suggested that batches of discs were exposed to atmospheric moisture for long periods, which could lead to degradation. Sources claimed that these same batches of antibiotics were also used in assays to check the discs were performing as claimed—because the antibiotics on the disc were from the same pack the tests wouldn’t be able to spot if the discs had gone off.

A follow up email, seen by the BMJ, by the same internal reviewer a year later (2007) suggests that these problems were still occurring.

A company project report from 2007 also shows environmental contamination in the production of the discs, so a disc containing one type of antibiotic might be contaminated with another. The report also said there was uncertainty about “the number of products that do not currently meet the definition of clean.”

A representative from Thermo Fisher, Oxoid’s parent company, told the BMJ that it has “robust procedures in place” for the proper handling of antibiotics. However, the representative added: “We have experienced isolated recalls due to the contamination of one antibiotic with another,” saying it has “established new procedures, reviewed and approved by the MHRA [UK Medicines and Healthcare Products Regulatory Agency], to ensure that this could not happen again.”

The amount of antibiotic that ends up on each disc can vary, so companies give a range for the potency of the disc. Although the US Food and Drug Administration states that discs must have 67-150% potency, the European Union does not specify a figure. AST discs fall under Europe’s medical device legislation. Oxoid’s website says that their AST discs are “produce[d] to the DIN [German Institute for Standardisation] specification, the tightest international standard.”5 To get the stamp of approval in Germany, discs should carry 90-125% of the stated antibiotic dose.

At a British Society of Antimicrobial Chemotherapy (BSAC) conference in 2005, Colin Booth, then vice president of science and technology at Oxoid and a senior member of BSAC, told a meeting about test methods: “We use the DIN standard, the titre standard 90-125%. Although within the specifications, we are generally at the high end of the range, so that the product is within the limits at the end of the shelf life.”6

But internal documents dating before and after Booth’s conference talk casts doubt on this claim. Far from showing that the discs fall into the upper range, these show that the company did not always know how much antibiotic was on the disc.

“The current concentration bioassays do not reliably indicate whether product is within 90%-125% of the stated concentration,” a 2007 internal project report says.

Questions over test procedures

Antibiotic discs are kept in the freezer. To check their quality, random samples are taken and sent to the laboratory to see if they contain the correct amount of antibiotic. There are two methods of testing—chemical (high performance liquid chromatography) and microbiological, which involves measuring the clear zone sizes around the discs with callipers. The results of the tests are then plotted on a graph showing known values to estimate the amount of antibiotic.

The company does not use chromatography for all discs. Indeed, the FDA considers microbiological testing sufficient. But the microbiological method Oxoid used to check the discs was not applied correctly.

In June 2003, an internal email from Booth admitted that the data from the tests were “flawed.” He didn’t know how accurate the discs were and speculated that they may give readings “plus or minus 25%” of what they should. “Clearly this is unacceptable,” he wrote. “There are many fixes for this problem, some of which are easy, some of which are going to hurt. I suggest we try the easy fix first,” he said. It is not clear what this “easy fix” was.

At this point the company changed some of the equipment used in the microbiology assay. However, internal documents and spreadsheets also suggest that they were using flawed methods that distorted data in two ways.

While the discs were still checked by measuring zone sizes, internal data analysis shows that the accuracy recorded was 10 times that achievable by the method used. In a microbiology plate assay Oxoid uses 18 test discs from the production batch. In many cases the zones around all 18 test discs were within 0.3 mm of each other, which is far beyond the accuracy achievable. This is both because there is variation between the discs (hence the range given for specification and the need to use 18 discs to test a batch) and because of the accuracy achievable by the human eye. However, when the results were internally checked analysts found that the range was, in fact, 1-2 mm.

In 2005, company employees approached a world authority on antimicrobial assays and an Oxoid consultant with the data and the methods they were concerned about. Copied into the correspondence was a member of senior management at Oxoid. It was also shared widely within the company.

The consultant said: “My immediate reaction is of suspicion. Is this too good to be true?”

In his analysis, he wrote: “I strongly suspect that zone size reading is a major problem.” He suspected there was “operator anticipation” of zone size.

These inaccurate zone sizes were then plotted to determine the concentration of antibiotics. Plots of bacterial response to antibiotics produce a curved graph, a point with which Oxoid agrees. However, Oxoid used a straight line on its graph, and did so for the full range of antibiotics.

The BMJ put it to Oxoid and Booth that the reason they used these methods for microbiological assay data was so that they met the product specification. Oxoid did not respond directly to the allegation, but a spokesperson told the BMJ that Oxoid “consistently reports accurate data” and “disagrees” with the view that its assay data are “too good to be true.”

Because the methods used to check the amount of antibiotic on the discs were flawed, the company did not know for certain how much antibiotic the discs contained—unless it used chromatography.

Failure to react

Despite these concerns, two years later (2007), an internal project report highlighted the problem still existed. “None of the bioassays currently used for routine analysis have (sic) been validated. Many of these methods are not fit for purpose. Poor quality data obtained from this type of analysis gives rise to a misleading indication of the quality of our products. This could lead to product failing QC [quality control] inspection when it is within specification or product passing when it should have been failed,” it says.

But even when chemistry tests did flag up problems, the company sometimes sent the discs out for use by hospitals around the world. One particular incident in 2005 shows how the company misled the UK regulator, the Medicines and Healthcare Products Regulatory Agency.

In September 2004 chromatography showed that a batch of discs labelled as containing cefpodoxime/clavulanic acid (10/1 μg), for use in Klebsiella and Escherischia coli infections, had “no quantifiable Cla [clavulanic acid] on them whatsoever.” However, the batch was cleared for sale.

An employee warned senior management that it should be considered “out-of-spec.” “I understand this batch is being sold, and thought you should look into it,” he wrote. But he was overruled and it was put onto the market regardless.

Only when Birmingham microbiologist Jenny Andrews—a senior figure in BSAC—spotted the problem a few months later (early 2005) and contacted Oxoid, did the company put out a recall alert instructing hospitals to discontinue the batch and consider retesting patients.7

In emails reporting what the MHRA had been told, seen by the BMJ, the company blamed a stability issue with the discs rather than admitting that prior data indicated that one component was completely missing. “The assumption is that the results were satisfactory when released but that there was deterioration on storage,” an internal company email said.

Oxoid told the BMJ that it is “confident that our products conform to all legal and regulatory requirements and that reports to regulators are accurate.”

Later emails also show that Oxoid would sometimes cherry pick the data to allow it to release batches. An email dated February 2006 from an Oxoid microbiologist said: “Had to delete the chemistry testing from this spec to allow the present batch to be passed.” It is not clear how widely this practice was adopted.

Once a batch is passed, it gets a certificate of analysis for release, which appears on the company’s website as a testimony to their rigour. “The information given is believed to be correct,” the certificate states.

However, the BMJ has been told that the company does not always enter all its chemistry assay results on the certificate—although Booth suggests that chemistry is more accurate.

An email dated April 2006 gives an example of this. “The certificate for the above lot is showing two results i.e. the micro and the HPLC assay. Please can you reload the results show[ing] only the plate assay [microbiology] is picked up,” a manager instructed her team.

Oxoid has stated that the DIN and FDA standards are microbiological standards and therefore it does not always release the chemical assay results. They have accepted that there are discrepancies between chemistry and biology results.

Emails also show that back in 2004, GSK asked Oxoid for discs containing mupirocin (a topical antibiotic effective against meticillin resistant Staphylococcus aureus) and clavulanic acid (used in combination with other antibiotics as it inhibits β-lactamase) for use in a trial. This combination of drugs is not currently on the market, but GSK was testing it.

An email from Booth told employees that they should ignore the chemistry results and release the batches regardless. “We need to resolve the position we find ourselves in with the product complying with the microbiological specification, but, failing to comply with the chemistry specification,” an email said. The solution was to: “amend the calculations for the antibiotic content to bring them within acceptable range.”

When the BMJ put this to Oxoid, a spokesperson said that everything was clearly relayed to GSK. “All of the methods and specifications were agreed to with GSK in writing. When a discrepancy between the chemical and biological results occurred, we communicated that to GSK and assured that the microbiological performance was satisfactory and acceptable to GSK,” he said.

He also told the BMJ that external audits by the US and UK regulators have shown that the company has complied with all standards of testing and data reporting—although FDA good practice guidance says that companies should not omit any results generated on a batch. Oxoid says it puts all results on the quality certificates.

Emails seen by the BMJ suggest that employees within the company have raised concerns about Oxoid’s practice, but they have been ignored. In April 2006, one employee argued against the company’s decision to average good and bad results to get a favourable result. Half of test samples in this batch failed. The FDA has told the BMJ that it expects all the discs in a batch to be uniform.

“The specification is a range and our customers are expecting each disc, rather than the average disc, to fall within this range,” the employee complained to the managers. The employee was also concerned that doing this masked the wide variability in how the discs were made: “I don’t understand how this is acceptable.”

Oxoid denied that it averages results and stated that it manufactures to the more stringent DIN standard rather than the FDA standard.

Blank discs

Others outside Oxoid also raised concerns. At a BSAC group meeting in 2005, one speaker asked if “it is possible to have blank discs in the cartridge” as he had generated a false “resistant” reading from an Oxoid disc.

Booth replied that although it was possible with old equipment, it was “very unlikely” with the company’s new equipment. Another clinician in the audience pushed further. “We have had this happen when six discs have been tested and one disc does not give the correct result.” Booth suggested that hospital clinicians may not be carrying out the test correctly.

However, sources have told the BMJ that far from being an isolated incidence, the new machine would often produce blank discs. Indeed, on 2 November 2012, Booth issued a warning through the MHRA that certain discs in vancomycin batches did not contain antibiotic.

Oxoid said that it is dealing with the problem in the full knowledge of the MHRA and that the blank discs are a “rare occurrence.”

Clinicians also told Andrews, who was responsible for helping to develop BSAC guidelines, that they were having problems with their discs. Oxoid’s response was that they were storing their discs incorrectly.

One concerned employee contacted Andrews in 2010 to alert her to the problems at the company.

However, she said she did “not want to get involved with this issue particularly as you [the employee] are using confidential material.” Andrews did not respond to an email asking her why she had done this.

Mark Wilcox, a consultant microbiologist at Leeds teaching hospitals, whose own study on Clostridium difficile testing kits found that the majority of them produced incorrect results, said it would be of “great concern” if any manufacturer failed to act on reports and queries from clinicians.

Internal criticisms

But the harshest criticism of all came in an internal process review conducted in May 2007 —two years after Oxoid suggested to the MHRA the recalled batch was an isolated incident. “Most if not all of the discs we make do not meet the stated specification . . . Poor quality data obtained from this type of analysis [microbiology] gives rise to a misleading indication of the quality of our products.”

Employees charged with the review said that this could lead to product failing when it is within specification or product passing when it should have been failed. “The lack of reliable data makes it impossible to effectively monitor our current manufacturing process,” it added.

That “lack of reliable data” may have direct knock-on effects in hospital laboratories. Clinicians accept there are a number of variables to control in a lab to achieve optimum results. But as Wilcox explains, you would expect discs to perform as certified by the manufacturer, so disc potency would not be high on the list of variables to investigate when problems arise.

Far from being a historical problem, a certificate of analysis for a disc containing voriconazole— an antifungal used to treat Candida infections— manufactured in February last year suggests that it contains 1.1 μg of the drug. However, chemistry results seen by the BMJ show that it in fact contained 1.44 μg.

In 2009, one employee became so disgruntled with Oxoid, they contacted the MHRA and supplied documentary evidence. On receiving no response, they contacted the regulator again in 2011.

In a letter to the MHRA seen by the BMJ, the employee said: “It was recognised by Oxoid in 2003 that the AST disc testing was invalid and often erroneous. As a consequence, Oxoid often manipulated the data in order to make the product saleable. In 2008 I sent the MHRA documentary proof of these practices. However, since that time there has been no significant change to the way Oxoid’s discs are tested.” The employee also informed the UK regulator that the company had “misled” it in the past.

On 29 June 2011, Kent Woods, chief executive of the MHRA, replied. He maintained that the MHRA had investigated his allegations. He stated that the company had implemented corrective actions in light of past problems and the MHRA had not received any further complaints through its adverse events reporting system.

“I can confirm that we did correspond with the company on the basis of your allegations with a view to satisfying ourselves that the product in question complied with the regulations,” Woods said.

The response has worrying implications for how well the MHRA acts on concerns of employees and that it is happy to take companies at their word. As the employee put it, the MHRA has “failed in its duty to protect the public.”

The BMJ asked the MHRA about its investigation. A spokesperson said that current European legislation requires any compliance investigation to be kept confidential: “This legislation is currently under review and we have made it clear to the European Commission that we believe that it is in the public interest that the European system is more transparent and information is easier to access.”

The spokesperson added: “We will quickly take action if new evidence emerges that any product does not meet European regulations. This action may include taking the product off the market or working with the manufacturer to ensure a product complies with European regulations.”

The same employee also contacted the FDA, supplying them with the necessary data. The BMJ asked the FDA if it had acted on the complaint. A spokesperson said: “While we cannot speak to any specific matter, we can tell you that when FDA receives information of this sort we generally follow up on it.”

External regulation

So what external certification system is in place for assuring the quality of AST discs? Like hip prostheses and breast implants, in vitro diagnostics—such as Oxoid’s AST discs and C difficile testing kits—have to be CE marked before they can be sold in Europe.

In the US, the discs are class II medical devices and have gone through the FDA’s 510(k) process. This allows a company to say that its device is similar to another device on the market, but it has to provide detailed evidence for this.

Under the in vitro diagnostics legislation in Europe, however, companies can self certify their products. This means they are able to put a CE mark on their product and say what it does without any external scrutiny.

The notified bodies—companies charged with assessing medical devices across Europe— just inspect factory processes to see if they are safe and produce products to a high standard. A CE certificate for manufacture is awarded and notified bodies then follow up with annual checks.

Indeed, Oxoid told the BMJ that: “External audits from 2003 through the present have never identified problems with the concentration of antimicrobial material on our discs.” Oxoid’s factory in Basingstoke was assessed by British notified body BSI in 2003 for the “design, development and manufacture of laboratory diagnostics for the detection, isolation, identification and susceptibility testing of micro-organisms.”

That same year, it also received approval to manufacture under the DIN specification from German notified body, DQS. Its good manufacturing practice certification was subsequently renewed in 2008.

The BMJ asked both notified bodies a series of questions. Did they ever spot problems when they audited the factory and were they aware of Oxoid’s internal reports? BSI refused to respond because the BMJ would not pass on the documents sent by the whistleblower.

“In the interest of all parties that transparency is played out and all parties are briefed on this issue, BSI would like to have full sight of the documentation (which could be redacted), so we can evaluate the bona fides and accuracy of the information, and then if necessary take such steps as may be required to address the allegations,” a spokesperson said.

DQS did not respond. According to Wilcox the method of regulating in vitro diagnostics needs to be improved. This is a “serious issue at present that needs addressing now, not in four or five years’ time,” he said.

Notes

Cite this as: BMJ 2013;346:f837

Footnotes

  • Competing interests: We have read and understood the BMJ Group policy on declaration of interests and have no relevant interests to declare.

  • Provenance and peer review: Commissioned; externally peer reviewed.

References