Studying new antibiotics for multidrug resistant infections: are today’s patients paying for unproved future benefits?

Multidrug resistance and development of new antibiotics

Although resistant infections have been commonplace since the advent of antibiotics in the 1940s, effective treatment still remains for most patients. Resistance should be of most concern when the drugs show decreased effects on morbidity and mortality. However, discussions about the rise of antibiotic resistance often focus on biomarkers of in vitro biological activity (the minimum inhibitory concentration (MIC) or pharmacodynamic parameters using MIC), assuming that these reflect patient outcomes.

Multidrug resistance is often described as decreased in vitro biological activity to multiple classes of drugs without considering remaining effective therapies. In fact, most patients do not have unmet medical needs since at least one effective therapy remains for them. When no effective treatment exists the unmet need is for a drug that is more effective (not equally or less effective) at reducing morbidity or mortality than current standards of care.5

Because smaller numbers of patients have no effective treatment option, premarketing trials of antibiotics usually recruit people with infections that remain susceptible to currently approved antibiotics.267 For trials in patients with serious infections, ethical standards require use of already approved antibiotics as control drugs since randomisation to placebo would expose patients to increased risk of illness or death. These trials can therefore assess either superior effectiveness or non-inferiority to existing standard of care.

Superiority versus non-inferiority hypotheses

Both overall and attributable mortality remain substantial for many serious infections that are susceptible to antibiotics, such as ventilator associated pneumonia.8 Nevertheless, it has been suggested that newly developed antibiotics are unlikely to prove superior to properly dosed comparators if the pathogen is susceptible to both agents.9 Thus, superiority trials in patients with effective options are deemed “not feasible.” Instead policy makers are pressured to accept less evidence of lower quality for regulatory approval of new antibiotics.10

Regulators in the US and EU seemingly have settled on non-inferiority trials as the main approach for developing evidence for approval, primarily based on a rationale that feasibility concerns are paramount. Use of such trials in populations that have effective existing treatment allows for lesser effectiveness in life threatening disease, while indirectly assuming effectiveness in future unstudied types of patients with serious infections that are resistant to all current drugs. This approach raises four serious scientific and ethical questions.

Does the trial ask the right questions?

The rationale for using non-inferiority hypotheses is well known: “New interventions may have little or no superiority to existing therapies, but, as long as they are not materially worse, may be of interest because they are less toxic, less invasive, less costly, require fewer doses, improve quality of life, or have some other value to patients.”111

But this rationale is rarely applied to investigational antibiotics. Instead the rationale is that the investigational antibiotics will have superior effectiveness in populations other than the one studied (that is, patients for whom the control drug is ineffective, who are excluded from non-inferiority trials), without expectation of non-efficacy benefits. Experience has shown that non-inferiority trials are not usually focused on the resistant organisms for which new therapeutic options are most needed.6 For antibiotic approvals from 1991 to 2011, only one of the 72 trial protocols or statistical analysis plans provided an explicit rationale for the acceptability of trade-offs between lesser efficacy and non-efficacy benefits.12 None of the trials delineated hypotheses or reported evaluation of non-efficacy benefits.

Rather than examining these trade-offs, investigators refer to trials as facilitating new antibiotics to market and providing incentives for drug companies.67 As one commentator stated, “the less demanding but admittedly indirect hurdle of noninferiority ... offers the potential to relatively rapidly identify drugs comparable in efficacy, jumpstarting antibiotic development.”13 But non-inferiority trials cannot show two interventions are exactly equal, and the results often do not rule out some level of harm because trials are often too small to show statistical inferiority.1

Are non-inferior drugs today going to be superior tomorrow?

The idea that drugs that are non-inferior in today’s patients will provide superior efficacy in future patients remains conjecture because the intended patient population has not been studied. Furthermore, there are compelling reasons to think the prediction is unlikely to come true.

Assumed future benefits are based on biomarkers of in vitro biological activity, termed an “improved microbiological spectrum of activity.”6 However, in vitro activity may not reflect direct patient benefits. Recent drugs with favourable in vitro activities compared with older drugs have been found to have increased mortality or decreased effectiveness in trials (table 1).

Patient factors are important. Those infected with resistant organisms are generally older, more critically ill, and have a greater incidence of renal insufficiency than patients with susceptible infections.26 These factors influence the effects of drugs on patient outcomes independent of in vitro factors.272829 Recently studied antibiotics in patients with susceptible infections have been less effective in patients with renal insufficiency, a group at greater risk of infection with resistant organisms.5 Drugs that may be slightly less effective in non-inferiority trials may be substantially less effective in more critically ill, unstudied patients.30 Heterogeneity of drug effects based on patient factors is common even in the absence of resistance.31

Finally, extrapolation to patients in whom the control intervention is not effective contradicts the basic premise of non-inferiority hypotheses and clouds interpretation of trial results. The design, analysis, and interpretation of non-inferiority results applies only to patients in whom the control intervention is effective—a principle called the constancy assumption.132

Less effective drugs can still promote resistance to other effective drugs, worsening the problem of resistance they were intended to address. Current sale and use of newly approved antibiotics may result in resistance as well, so they may not be effective in the future, and resistance has been noted soon after approval of new antibiotics.33 But it is unlikely that manufacturers will delay use of new drugs until current ones are less effective since companies have limited periods of market exclusivity and business principles favour maximising current over future earnings.34

How well have antibiotics approved after non-inferiority trials performed?

Although the current antibiotics of last resort were approved after non-inferiority trials,7 the full track record of three decades of antibiotic approvals, primarily on non-inferiority trials, is less favourable. Of 61 antibiotics approved between 1980 and 2009, 26 (43%) were withdrawn by 2013, many for poor sales or safety and effectiveness problems.43 None were withdrawn because of antibiotic resistance despite increasing in vitro resistance.

In recent years, payers have resisted premium pricing for antibiotics approved on the basis of non-inferiority without evidence of other advantages over currently available options. The Centers for Medicare and Medicaid Services, which provides health insurance to around 100 million Americans, denied such premium pricing for dalbavancin, citing lack of clinical evidence of substantial improvement in effectiveness compared with older drugs or that improved convenience affected patient outcomes.5

Some people have suggested that we should respond to the inadequate reimbursement for antibiotics by allowing regulatory agencies to approve new antibiotics with less evidence at earlier stages of development.44 The US 21st Century Cures Act,45 signed in December 2016, gave the FDA the green light to label new drugs for patients with “limited or no options.” 1546 But it is hard to see how this would assuage payers’ scepticism about premium prices, given that drugs with promising preclinical and early clinical data can fail to show benefits for patients in later stage trials.47

Recent examples of antibiotics in which promising hypothesis generating information failed to predict patient outcomes in randomised trials show the challenges in relying on preclinical and early clinical data (table 1). The PROVIDE study, an observational study to validate the pharmacodynamic index for vancomycin among patients with meticillin resistant Staphylococcus aureus bloodstream infection, found that optimised doses were associated with increased harm but no added efficacy.48 A recent review showed a lack of evidence that pharmacodynamic dosing of antibiotics resulted in better outcomes in nosocomial pneumonia.49

Moving forward: superiority trials and innovative trial designs

Only superiority hypotheses can determine whether new antibiotics meet the medical need of patients with multidrug resistant infections who currently have no treatment options. Superiority trials need not be large, nor impossible to recruit.79 The Centers for Disease Control and Prevention estimates that two million people annually are infected with resistant pathogens in the US alone, leading to around 23 000 deaths.50 It notes “many more die from other conditions that were complicated by an antibiotic resistant infection.” These numbers should be a sufficient source of patients to study. If these populations are difficult to find and enrol, then we should enhance investment in clinical trial infrastructure in locations where resistance is currently prevalent, such as the Multi-Drug Resistant Organism (MDRO) Network, a collaboration within the National Institutes of Health funded Antibacterial Resistance Leadership Group (ARLG). Another approach to increase participants is to conduct the study on multiple sites.51 Pre-enrolment strategies such as early informed consent in patients at risk of resistant infections may also aid enrolment.

Superiority of effective investigational drugs can be shown with few enrolled participants in resistant pathogen settings since death rates are higher. The ARLG recently conducted an observational study in around 140 patients (compared with 500-600 patients used in non-inferiority trials) that suggested improved mortality of a new antibiotic compared with an older antibiotic52; this could be confirmed in small randomised trials. As few as 24 enrolled patients would be needed for a well conducted trial of a new drug that decreases mortality with similar efficacy to penicillin relative to placebo in early trials: if 9/12 patients survive with the new drug compared with 4/12 with standard of care P=0.03.³¹

In addition, large safety databases are not required when the benefit is improved survival in patients with no options. Superiority trials are also not subject to restrictions that decrease the feasibility of non-inferiority trials, such as standardised control drugs of known effectiveness, exclusion of patients taking previously effective therapies, and replication of design features of earlier studies.

The existence of effective therapies does not mandate non-inferiority hypotheses. Superiority designs can use active controls. In randomised, blinded “add-on” superiority trials, new drugs are added to best standard therapy and compared with standard therapy plus placebo. This design is used in oncology, where drug resistance also is common.5354 Interventions with novel mechanisms of action such as host immune modifiers would have to be evaluated in add-on superiority trials with conventional antibiotics. The focus on non-inferiority trials could therefore delay appropriate study and evaluation of such novel interventions.

Superiority designs also might catalyse development of diagnostics to streamline trial enrolment and facilitate appropriate patient selection and antibiotic use in clinical practice. Non-inferiority trials reinforce the lack of incentive to develop appropriate diagnostics since inaccurate diagnosis can minimise observed differences between interventions, making non-inferiority more likely.55 The emphasis on identifying biomarkers for superiority trials in oncology has helped select patients who benefit and avoid patients at high risk of toxicity.56

Innovative trial designs and analyses can avoid the drawbacks of non-inferiority trials while addressing clinically relevant pragmatic questions about superiority of interventions. The ARLG is using methods such as the desirability of outcome ranking (DOOR) and partial credit design strategies, which combine benefits and harms to evaluate overall superiority.5758

We share concern about the growing need for more effective treatments for infectious diseases caused by multidrug resistant bacteria. But new treatments should show demonstrable benefits for patients, not just activity against organisms, and the goal of drug development should be to help both current and future patients. Regulators should therefore insist on properly designed, executed, and powered superiority trials in patients who lack any effective options and in whom the benefit-risk considerations are justifiable.