Feature Medical Research

Creating credibility and value in comparative effectiveness research

BMJ 2012; 345 doi: http://dx.doi.org/10.1136/bmj.e7302 (Published 01 November 2012) Cite this as: BMJ 2012;345:e7302
  1. Bob Roehr, freelance journalist
  1. 1Washington, DC, USA
  1. BobRoehr{at}aol.com

Bob Roehr examines the difficulties of conducting research to help doctors and patients make better treatment decisions

The unsustainable rise in healthcare costs and the transition from fee for service reimbursement to payments that are capitated, bundled, or based on outcomes is generating renewed interest in comparative effectiveness research across the spectrum of healthcare stakeholders in the United States.

However, such research, which compares existing healthcare interventions to determine which work best for which patients and which pose the greatest benefits and harms, will be accepted and integrated into practice only if there is consensus on what a study’s results truly mean, according to a cross section of experts.

Failure to reach consensus on what a study has the potential to show beforehand can fuel ongoing controversy, such as those over the timing and frequency of mammography in screening for breast cancer and interventions for prostate cancer and back pain, they said.

This discussion grew out of a recent exercise and series of papers organized by the journal Health Affairs.1 2 The exercise shows the difficulty of achieving agreement after the fact, and points to how the processes of comparative effectiveness research must change if it is to be embraced.

The exercise centered on a fictional study: a retrospective analysis of electronic health records comparing those who did not receive treatment with those who received two drugs for migraine that had different trade-offs in efficacy, ease of use, adherence, side effects, and cost.

The hypothetical study fell short on being patient centered in three important ways, said Rachael Fleurence, a scientist with the newly established Patient-Centered Outcomes Research Institute (PCORI). The quasi-governmental agency was created as part of the US Affordable Care Act to advance comparative effectiveness research.

Fleurence said that the mock study did not generate information relevant to patients making healthcare decisions. It lacked subgroup analysis along the lines of sex, age, and other criteria.

There also did not seem to be a real demand by patients for the type of information on usage and cost generated by the study, and so it seemed unlikely to have much effect on their choice of drug. Finally, Fleurence said, the method used resulted in a selection bias that meant patients in the untreated group were likely to have had milder symptoms.

“Despite some trappings of quality, the study is flawed,” said Aaron S Kesselheim, assistant professor of medicine at Harvard Medical School. It was not a randomized study, and patients had been on treatment, perhaps already identifying what worked best for them through trial and error. Nor was it clear that all confounding factors were controlled for.

“These types of biases are common in observational research,” he said. Furthermore, “The large electronic databases make it possible to design and conduct dozens of observational analyses, adjusting design and covariant elements until one obtains the results that one wants.”

The potential for abuse, coupled with industry’s history of manipulating data and hiding negative studies, along with its misleading educational and marketing practices is why “the FDA [Food and Drug Administration] is appropriately wary” of permitting drug companies to promote the findings of comparative effectiveness research, Kesselheim said.

While the FDA has the legal authority to allow the use of such research in promotional activities, it has not developed a consensus on what distinguishes good and poor observational studies, he added.

“Companies want safe harbors,” of what the FDA will allow, said Tevi Troy, a senior fellow at the Hudson Institute and former FDA official, and then they can figure out what they can do without technically violating the regulation. “The FDA doesn’t always want to provide that,” particularly in an area that is rapidly changing and without compelling pressure from other stakeholders to develop those regulations.

Today the randomized controlled trial is taken as the “gold standard” of clinical research. Kesselheim said it was important to remember that it took decades for consensus to develop around the standards and validity of that process, particularly when it generated findings that flew in the face of what individual clinicians might have observed in practice. It will take time to develop similar standards and confidence in observational studies.

Modern patient

There has been a revolution in what it means to be a patient. A generation ago patients were isolated, largely uninformed, and disempowered. Today they have instant access to much of the same information as their physician; many are networked with and learn from fellow patients; and disease groups are large in number, better organized, and more aggressive in advancing the needs of patients.

“This really turns the research paradigm on its head,” said Marc Boutin, executive vice president of the National Health Council, an umbrella organization of patient advocacy organizations.

He said surveys had found a bit of a contradictory reaction to comparative effectiveness research among patients. On the one hand, they often believe that it already is being done and integrated into healthcare; on the other, many fear it is going to be used to deny them access to care. Those fears were exploited with political rhetoric of “death panels” during the debate on healthcare reform.

Comparative effectiveness research is an opportunity to get questions answered in a way that will help patients make decisions on their own healthcare, according to Boutin. “We want to know what is the most effective in the context of our lives.” The answer may differ from patient to patient depending on how they value each factor associated with an intervention.

Comparative effectiveness research “is the application, not just the generation of knowledge,” he said. Although methodological standards must be addressed, “you also need an independent evaluation of the results to determine if they are applicable for certain purposes,” such as creation of treatment guidelines and making individual decisions.

“The application requires professional standards,” that are agreed to by all stakeholders, he said. “We can’t simply focus on industry,” Boutin added. “Every stakeholder has a bias. We all want the information; we want it for different reasons. That’s okay. But everybody has to be at the table” for all steps of the research process, starting with formulating the research questions so that they generate data that are useful in making treatment decisions.

He said patient groups would not disseminate information if they did not think it was credible. This flow of information, the reputation of an intervention and its sponsors, can have a significant influence on patients’ willingness to participate in a clinical trial or use an approved product.

Arthur Caplan, an ethicist with the New York University Langone Medical Center, commented, “Most of the comparative effectiveness research that I see is not user friendly. It needs to be translatable into understandable information.”

The growing number of retractions of papers because of fraud and plagiarism3 has undercut some of the authority of peer review journals among both patients and physicians, he said. Caplan also fears the growing number of medical publications threatens to water down the peer review process.

He wonders whether the process can continue to operate on a voluntary basis. He suggested, “Journals may need to hire more statistical help in analyzing what comes in; it sometimes is beyond the peer reviewer’s competence and time.” Standards and training for peer reviewers also might be advisable.

Changing medical practice

The pharmaceutical industry has been hammered for its promotional practices and has begun to cut back on those activities.4 It is going through a wrenching readjustment as a generation of blockbuster drugs that created enormous profits are coming off patent and few replacements are in sight.

The companies have never been enthusiastic about comparative effectiveness research, perhaps because, as we have learnt with the increasingly visible record of antidepressants, they knew the drugs didn’t work very well. Now, facing a diminished revenue stream, they are unlikely to pour money into this type of research.

PCORI has the potential to fill this gap and more. But it is funded only through to 2019. It has also drawn the wrath of some conservatives, who oppose it on ideological grounds. And in a period of austerity, members of both parties may see it as a source of money to maintain levels of spending for healthcare programs with more established constituencies. And, as Caplan noted, “The public level of trust in messages from government is not super high just now.”

“Data alone is inadequate” to change behavior, including the practice of medicine, said Michael A Fischer, associate professor of medicine at Harvard Medical School. It also takes education, and often incentives and penalties, which is why it took 15-20 years for use of β blockers to become the near universal standard of care.

“The model we have had for communicating information, where industry dominated the flow of information, has proved to be very problematic,” said Fischer. He supports the concept of non-commercial “academic detailing” as an alternative to the often biased education and marketing activities conducted by drug companies.

Robert W Dubois, chief science officer for the National Pharmaceutical Council, called academic detailing “a very interesting experiment.” However, he lamented that measurement of those activities currently focuses on physician acceptance—it does not measure whether physicians change their practice or patient outcomes are improved.

Perhaps the more important driver in adopting comparative effectiveness research and applying it to clinical practice in the United States will be the increased integration of the delivery of care. Systems require standardization, and organizations use that foundation of standardization to conduct training, establish preferred practices or formularies, and develop more sophisticated electronic health record software that supports decision making at the point of care.

Large healthcare systems like Kaiser Permanente conduct their own comparative effectiveness research. When two drugs have similar efficacy, they make the preferred drug the one for which they can negotiate the best price. This integration can achieve better population based outcomes as well as reduce overall costs.


Cite this as: BMJ 2012;345:e7302


  • Competing interests: The author has completed the ICMJE unified disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declares no support from any organisation for the submitted work; no financial relationships with any organisation that might have an interest in the submitted work in the previous three years; BR is an independent biomedical journalist who has written reports and other documents, either directly or as a subcontractor, for non-profit clients including the American Association for the Advancement of Science and the US National Institutes of Health.

  • Provenance and peer review: Commissioned; not externally peer reviewed.