Studying new antibiotics for multidrug resistant infections: are today’s patients paying for unproved future benefits?BMJ 2018; 360 doi: https://doi.org/10.1136/bmj.k587 (Published 22 February 2018) Cite this as: BMJ 2018;360:k587
- John H Powers, professor of clinical medicine1,
- Scott R Evans, senior research scientist2,
- Aaron S Kesselheim, associate professor of medicine3
- 1George Washington University School of Medicine, Washington, DC, USA
- 2Department of Biostatistics and the Center for Biostatistics in AIDS Research, Harvard T H Chan School of Public Health, Boston, MA, USA
- 3Program On Regulation, Therapeutics, And Law (PORTAL), Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women’s Hospital, Boston, MA
- Correspondence to: J H Powers
Concerns over the threat of antibiotic resistance have fuelled legislative and regulatory efforts to promote development of new treatments for serious infections. Fears arise that the world could face serious, multidrug resistant bacterial diseases that lack effective treatments. An optimal approach would be to develop drugs that improve on current effective therapies for today’s patients, while potentially offering benefits for future patients as antibiotic resistance evolves.
Unlike in many other therapeutic areas, however, most new antibiotics are approved without evidence of superior efficacy in clinical trials. Rather, regulatory authorities increasingly permit “non-inferiority” hypotheses. These allow lesser effectiveness than already approved drugs in trade-off for other benefits such as decreased adverse effects.12345 Indeed, some claim that superiority hypotheses are unethical for investigational antibiotics since they might expose patients in the control group to harm.6
A review evaluating US Food and Drug Administration approvals from 2009 to 2015 showed eight new antibiotics indicated for “serious and life threatening” diseases, with seven approved solely on evidence of non-inferiority. Recent FDA guidance supports extrapolating non-inferiority results in patients with effective treatments to presume superior outcomes in patients without effective options.25 But are drugs that are “non-inferior” today going to be superior tomorrow? Do non-inferiority hypotheses expose today’s patients with treatable life threatening infections to increased risk of harm by allowing investigational antibiotics with less effectiveness onto the market?
Multidrug resistance and development of new antibiotics
Although resistant infections have been commonplace since the advent of antibiotics in the 1940s, effective treatment still remains for most patients. Resistance should be of most concern when the drugs show decreased effects on morbidity and mortality. However, discussions about the rise of antibiotic resistance often focus on biomarkers of in vitro biological activity (the minimum inhibitory concentration (MIC) or pharmacodynamic parameters using MIC), assuming that these reflect patient outcomes.
Multidrug resistance is often described as decreased in vitro biological activity to multiple classes of drugs without considering remaining effective therapies. In fact, most patients do not have unmet medical needs since at least one effective therapy remains for them. When no effective treatment exists the unmet need is for a drug that is more effective (not equally or less effective) at reducing morbidity or mortality than current standards of care.5
Because smaller numbers of patients have no effective treatment option, premarketing trials of antibiotics usually recruit people with infections that remain susceptible to currently approved antibiotics.267 For trials in patients with serious infections, ethical standards require use of already approved antibiotics as control drugs since randomisation to placebo would expose patients to increased risk of illness or death. These trials can therefore assess either superior effectiveness or non-inferiority to existing standard of care.
Superiority versus non-inferiority hypotheses
Both overall and attributable mortality remain substantial for many serious infections that are susceptible to antibiotics, such as ventilator associated pneumonia.8 Nevertheless, it has been suggested that newly developed antibiotics are unlikely to prove superior to properly dosed comparators if the pathogen is susceptible to both agents.9 Thus, superiority trials in patients with effective options are deemed “not feasible.” Instead policy makers are pressured to accept less evidence of lower quality for regulatory approval of new antibiotics.10
Regulators in the US and EU seemingly have settled on non-inferiority trials as the main approach for developing evidence for approval, primarily based on a rationale that feasibility concerns are paramount. Use of such trials in populations that have effective existing treatment allows for lesser effectiveness in life threatening disease, while indirectly assuming effectiveness in future unstudied types of patients with serious infections that are resistant to all current drugs. This approach raises four serious scientific and ethical questions.
Does the trial ask the right questions?
The rationale for using non-inferiority hypotheses is well known: “New interventions may have little or no superiority to existing therapies, but, as long as they are not materially worse, may be of interest because they are less toxic, less invasive, less costly, require fewer doses, improve quality of life, or have some other value to patients.”111
But this rationale is rarely applied to investigational antibiotics. Instead the rationale is that the investigational antibiotics will have superior effectiveness in populations other than the one studied (that is, patients for whom the control drug is ineffective, who are excluded from non-inferiority trials), without expectation of non-efficacy benefits. Experience has shown that non-inferiority trials are not usually focused on the resistant organisms for which new therapeutic options are most needed.6 For antibiotic approvals from 1991 to 2011, only one of the 72 trial protocols or statistical analysis plans provided an explicit rationale for the acceptability of trade-offs between lesser efficacy and non-efficacy benefits.12 None of the trials delineated hypotheses or reported evaluation of non-efficacy benefits.
Rather than examining these trade-offs, investigators refer to trials as facilitating new antibiotics to market and providing incentives for drug companies.67 As one commentator stated, “the less demanding but admittedly indirect hurdle of noninferiority ... offers the potential to relatively rapidly identify drugs comparable in efficacy, jumpstarting antibiotic development.”13 But non-inferiority trials cannot show two interventions are exactly equal, and the results often do not rule out some level of harm because trials are often too small to show statistical inferiority.1
Are non-inferior drugs today going to be superior tomorrow?
The idea that drugs that are non-inferior in today’s patients will provide superior efficacy in future patients remains conjecture because the intended patient population has not been studied. Furthermore, there are compelling reasons to think the prediction is unlikely to come true.
Assumed future benefits are based on biomarkers of in vitro biological activity, termed an “improved microbiological spectrum of activity.”6 However, in vitro activity may not reflect direct patient benefits. Recent drugs with favourable in vitro activities compared with older drugs have been found to have increased mortality or decreased effectiveness in trials (table 1).
Patient factors are important. Those infected with resistant organisms are generally older, more critically ill, and have a greater incidence of renal insufficiency than patients with susceptible infections.26 These factors influence the effects of drugs on patient outcomes independent of in vitro factors.272829 Recently studied antibiotics in patients with susceptible infections have been less effective in patients with renal insufficiency, a group at greater risk of infection with resistant organisms.5 Drugs that may be slightly less effective in non-inferiority trials may be substantially less effective in more critically ill, unstudied patients.30 Heterogeneity of drug effects based on patient factors is common even in the absence of resistance.31
Finally, extrapolation to patients in whom the control intervention is not effective contradicts the basic premise of non-inferiority hypotheses and clouds interpretation of trial results. The design, analysis, and interpretation of non-inferiority results applies only to patients in whom the control intervention is effective—a principle called the constancy assumption.132
Less effective drugs can still promote resistance to other effective drugs, worsening the problem of resistance they were intended to address. Current sale and use of newly approved antibiotics may result in resistance as well, so they may not be effective in the future, and resistance has been noted soon after approval of new antibiotics.33 But it is unlikely that manufacturers will delay use of new drugs until current ones are less effective since companies have limited periods of market exclusivity and business principles favour maximising current over future earnings.34
When is it ethical to expose clinical trial participants to potential increased harm?
Non-inferiority trials use current patients with effective options as surrogates for future patients with no effective options. But the benefit-risk assessments in these two populations should differ substantially, since patients with no options may be willing to accept more risk of harm than patients who have effective options. This raises ethical questions regarding beneficence and justice in the selection of research participants for non-inferiority trials and has implications for informed consent.1235
The Declaration of Helsinki—the basis for modern ethical principles related to human subjects’ experimentation—calls for caution not just in placebo controlled trials but also when exposing patients to interventions less effective than current standards of care. It states, “Patients who receive any intervention less effective than the best proven one, placebo, or no intervention will not be subject to additional risks of serious or irreversible harm as a result of not receiving the best proven intervention.”36
The FDA also has stated “it may be important to consider whether a new product is less effective than available alternatives therapies, when less effectiveness could present a danger to the patient or to the public,” citing examples of life threatening and contagious diseases.3738 Similarly, the European Medicines Agency stated, “Where the treatment under consideration is used for the prevention of death or irreversible morbidity and there is no second chance for treatment it can be very difficult to justify a non-inferiority margin of any size.”39
Non-inferiority hypotheses are ethical in some situations (box 1). However, they are not ethical for trials of treatments for acute, life threatening infections that have effective therapies. Such patients may not find it acceptable to trade reduced effectiveness for fewer adverse effects or improved convenience. This is particularly true if lesser effectiveness translates to increased deaths or serious morbidity, regardless of whether these are the primary endpoints of the trial. Allowing rescue therapy for participants who do not respond to study interventions is a theoretical possibility, but it may not be achievable because of rapid disease progression in acute life threatening infections. Some have suggested that any delay in giving effective antibiotics increases mortality.4142
Appropriate use of non-inferiority hypotheses
Non-inferiority trials use hypotheses in which investigators aim to rule out a chosen amount of inferior effectiveness of a new intervention compared with an older effective intervention.1
Non-inferiority trials are not designed to confirm that two interventions are equal, “not worse” by any amount, or “as good as” each other, because ruling out any amount of inferior effectiveness requires demonstration of superiority.
Older reliably effective therapy exists and these effects can be quantified
The effect of older therapy on morbidity or mortality is so important to patients that they cannot be ethically randomised to placebo
There are hypothesised benefits of new interventions other than improved effectiveness
Potential decrements in effectiveness are based on patient input and are ethically acceptable when they do not place patients at risk for increased irreparable harm.36
An example is a new drug for treatment of self resolving, uncomplicated urinary tract infections, in which placebo controlled trials reliably and reproducibly show that older antibiotics shorten patient symptoms,29 and which may have fewer adverse effects than older effective agents. Patients may therefore be willing to accept the risk of small amounts of inferior effectiveness (eg, an additional day to achieve resolution of symptoms) without risk of serious irreparable harm in an overall favourable trade-off.40RETURN TO TEXT
Patient input can help define the boundaries of acceptable trade-offs for non-inferiority hypotheses, but this input hasn’t happened over the past two decades.12 International guidance cautions against allowing greater losses of effectiveness with new interventions in non-inferiority trials solely to decrease the required number of enrolled participants.32
How well have antibiotics approved after non-inferiority trials performed?
Although the current antibiotics of last resort were approved after non-inferiority trials,7 the full track record of three decades of antibiotic approvals, primarily on non-inferiority trials, is less favourable. Of 61 antibiotics approved between 1980 and 2009, 26 (43%) were withdrawn by 2013, many for poor sales or safety and effectiveness problems.43 None were withdrawn because of antibiotic resistance despite increasing in vitro resistance.
In recent years, payers have resisted premium pricing for antibiotics approved on the basis of non-inferiority without evidence of other advantages over currently available options. The Centers for Medicare and Medicaid Services, which provides health insurance to around 100 million Americans, denied such premium pricing for dalbavancin, citing lack of clinical evidence of substantial improvement in effectiveness compared with older drugs or that improved convenience affected patient outcomes.5
Some people have suggested that we should respond to the inadequate reimbursement for antibiotics by allowing regulatory agencies to approve new antibiotics with less evidence at earlier stages of development.44 The US 21st Century Cures Act,45 signed in December 2016, gave the FDA the green light to label new drugs for patients with “limited or no options.” 1546 But it is hard to see how this would assuage payers’ scepticism about premium prices, given that drugs with promising preclinical and early clinical data can fail to show benefits for patients in later stage trials.47
Recent examples of antibiotics in which promising hypothesis generating information failed to predict patient outcomes in randomised trials show the challenges in relying on preclinical and early clinical data (table 1). The PROVIDE study, an observational study to validate the pharmacodynamic index for vancomycin among patients with meticillin resistant Staphylococcus aureus bloodstream infection, found that optimised doses were associated with increased harm but no added efficacy.48 A recent review showed a lack of evidence that pharmacodynamic dosing of antibiotics resulted in better outcomes in nosocomial pneumonia.49
Moving forward: superiority trials and innovative trial designs
Only superiority hypotheses can determine whether new antibiotics meet the medical need of patients with multidrug resistant infections who currently have no treatment options. Superiority trials need not be large, nor impossible to recruit.79 The Centers for Disease Control and Prevention estimates that two million people annually are infected with resistant pathogens in the US alone, leading to around 23 000 deaths.50 It notes “many more die from other conditions that were complicated by an antibiotic resistant infection.” These numbers should be a sufficient source of patients to study. If these populations are difficult to find and enrol, then we should enhance investment in clinical trial infrastructure in locations where resistance is currently prevalent, such as the Multi-Drug Resistant Organism (MDRO) Network, a collaboration within the National Institutes of Health funded Antibacterial Resistance Leadership Group (ARLG). Another approach to increase participants is to conduct the study on multiple sites.51 Pre-enrolment strategies such as early informed consent in patients at risk of resistant infections may also aid enrolment.
Superiority of effective investigational drugs can be shown with few enrolled participants in resistant pathogen settings since death rates are higher. The ARLG recently conducted an observational study in around 140 patients (compared with 500-600 patients used in non-inferiority trials) that suggested improved mortality of a new antibiotic compared with an older antibiotic52; this could be confirmed in small randomised trials. As few as 24 enrolled patients would be needed for a well conducted trial of a new drug that decreases mortality with similar efficacy to penicillin relative to placebo in early trials: if 9/12 patients survive with the new drug compared with 4/12 with standard of care P=0.03.31
In addition, large safety databases are not required when the benefit is improved survival in patients with no options. Superiority trials are also not subject to restrictions that decrease the feasibility of non-inferiority trials, such as standardised control drugs of known effectiveness, exclusion of patients taking previously effective therapies, and replication of design features of earlier studies.
The existence of effective therapies does not mandate non-inferiority hypotheses. Superiority designs can use active controls. In randomised, blinded “add-on” superiority trials, new drugs are added to best standard therapy and compared with standard therapy plus placebo. This design is used in oncology, where drug resistance also is common.5354 Interventions with novel mechanisms of action such as host immune modifiers would have to be evaluated in add-on superiority trials with conventional antibiotics. The focus on non-inferiority trials could therefore delay appropriate study and evaluation of such novel interventions.
Superiority designs also might catalyse development of diagnostics to streamline trial enrolment and facilitate appropriate patient selection and antibiotic use in clinical practice. Non-inferiority trials reinforce the lack of incentive to develop appropriate diagnostics since inaccurate diagnosis can minimise observed differences between interventions, making non-inferiority more likely.55 The emphasis on identifying biomarkers for superiority trials in oncology has helped select patients who benefit and avoid patients at high risk of toxicity.56
Innovative trial designs and analyses can avoid the drawbacks of non-inferiority trials while addressing clinically relevant pragmatic questions about superiority of interventions. The ARLG is using methods such as the desirability of outcome ranking (DOOR) and partial credit design strategies, which combine benefits and harms to evaluate overall superiority.5758
We share concern about the growing need for more effective treatments for infectious diseases caused by multidrug resistant bacteria. But new treatments should show demonstrable benefits for patients, not just activity against organisms, and the goal of drug development should be to help both current and future patients. Regulators should therefore insist on properly designed, executed, and powered superiority trials in patients who lack any effective options and in whom the benefit-risk considerations are justifiable.
New antibiotics for life threatening infections are approved on non-inferiority hypotheses, which allow for lesser effectiveness
It cannot be assumed that non-inferior effectiveness today will translate to future superior effectiveness
When effective treatment exists, patients should not be exposed to increased risk of irreparable harm merely because they are easier to enrol in trials
Superiority trials are feasible and more ethical in patients with resistant infections who have no treatment options
Compared with non-inferiority trials they require smaller patient numbers, have no restrictions on previous treatment, and can include patient relevant endpoints
We thank Paul Sax, Ursula Theuretzbacher, Charles Natanson, and Irene Cortes Puch for their comments on previous drafts.
Competing interests: We have read and understood BMJ policy on declaration of interests and declare the following: ASK is a member of the Collaborative Research Programme in Biomedical Innovation Law at the University of Copenhagen (which is supported by grant NNF17SA027784 from the Novo Nordisk Foundation) and is funded by the Laura and John Arnold Foundation, with additional funding from the Harvard Program in Therapeutic Science and the Engelberg Foundation. He has received grants from the FDA Office of Generic Drugs and Division of Health Communication (2013-16). SRE is a member of the Antibacterial Resistance Leadership Group funded by the National Institute of Allergy and Infectious Diseases, National Institutes of Health; has served as a consultant for Roche, Pfizer, Takeda, Novartis, Merck, Achaogen, Cubist, GSK, Amgen, Boehringer-Ingelheim, Sunnovion, Taris, Alcon, Chelsea, Mannkind, QRx Pharma, Genentech, Auspex, Affymax, and FzioMed; and has received honorariums from the American Statistical Association, the Society for Clinical Trials, the FDA, the NIH, Osaka University, the National Cerebral and Cardiovascular Center of Japan, the Drug Information Association, the Huntington’s Study Group, the Muscle Study Group, the NJMS / Rutgers, the University of Rhode Island, Boston University, IMMPACT, PPRECISE, DeGruyter, Massachusetts General Hospital, and the City of Hope. JHP has served as a consultant for Abbvie, Cardeas, Cempra, Contrafect, Gilead, Johnson and Johnson, Lilly, MedImmune, Otsuka, and Roche and is an analysis adviser for The BMJ. All authors have contributed equally to the concepts, drafting, and review of this manuscript
Provenance and peer review: Not commissioned; externally peer reviewed.