Intended for healthcare professionals

Analysis

Endorsing performance measures is a matter of trust

BMJ 2018; 360 doi: https://doi.org/10.1136/bmj.k703 (Published 23 February 2018) Cite this as: BMJ 2018;360:k703

This article has a correction. Please see:

  1. Dharmvir S Jaswal, staff physician,
  2. Charles Natanson, senior investigator,
  3. Peter Q Eichacker, senior investigator
  1. Critical Care Medicine Department, Clinical Center, National Institutes of Health, Bethesda, MD 20892, USA
  1. Correspondence to PQ Eichacker peichacker{at}mail.cc.nih.gov

Healthcare agencies should ensure that performance measures for accreditation or reimbursement are supported by sound scientific evidence and safeguarded from potential commercial influences, argue Peter Eichacker and colleagues

Key messages

  • The US National Quality Forum endorses performance measures that the Centers for Medicare and Medicaid Services can then adopt nationally

  • The forum’s endorsement of a bundle of performance measures for managing sepsis, including the need for central venous pressure and blood oxygen saturation measures, failed to adhere to the agency’s own standards regarding the soundness of supporting scientific evidence and freedom from potential commercial influences

  • Unless government can ensure that its endorsements adhere to these standards, trust in the performance measures it endorses will be undermined

  • Commercial representatives should have no role in endorsing performance measures

Using evidence based performance measures to guide patient treatment is advocated to improve outcomes and decrease costs.1 The US Centers for Medicare and Medicaid Services uses performance measures to regulate US healthcare by withholding reimbursement and accreditation from non-compliant providers.234 With such compelling incentives, the performance measures essentially become healthcare mandates.

The National Quality Forum evaluates and endorses performance measures for the Centers for Medicare and Medicaid Services, ensuring that they are supported by strong, unbiased, scientific evidence.5 The NHS’s Quality and Outcome Framework has a similar role.67

SEP-1 is a performance measure for managing patients with sepsis that was endorsed by the forum and adopted by the Centers for Medicare and Medicaid Services while controversial interventions were still undergoing clinical testing. Even after this testing showed lack of efficacy, the interventions were not removed. Furthermore, a trade organisation with direct ties to a manufacturer standing to benefit from SEP-1 took part in the endorsement. Without rigorous, independent scientific review protected from potential industry influence, endorsements like that of SEP-1 will undermine trust in governmental agencies and compromise patient care.

National Quality Forum endorsement process

The forum is a non-profit, non-partisan agency founded in 1999, with member organisations including medical societies, healthcare corporations, insurers, trade associations, and manufacturers. It publicly requests the voluntary submission of performance measures for selected areas of healthcare and then determines through consensus whether they meet the agency’s endorsement criteria (box 1).89101112

Box 1

Summary of the National Quality Forum’s endorsement process

A steering committee of 20-25 people, who can be nominated by and from member organisations, is central to the endorsement process.8 Nominees cannot have any financial or other interest that could “actually, or be perceived to, significantly impede (the nominee’s) objectivity or create an unfair competitive advantage for (the nominee) or organization associated with (the nominee).”8 For approval there must be “unambiguous evidence” that the performance measure improves health.9 Randomised controlled trials providing precise, unbiased effect estimates are deemed high quality evidence. Those with bias flaws or non-randomised trials with imprecise effect estimates or that do not control for confounders are considered low quality evidence. For composite measures, the forum requires that each component intervention meets the criteria for benefit.10 Performance measures approved by the committee are submitted for public comment and then evaluated by the forum’s member organisations and the Consensus Standards Approval Committee. The forum’s board of directors decides on final endorsement.8 More than 300 performance measures have been endorsed by the National Quality Forum and adopted by the Centers for Medicare and Medicaid Services.1112

RETURN TO TEXT

Bundle of performance measures for sepsis

In 2008 the forum endorsed a performance measure for managing sepsis submitted by Henry Ford Hospital (table 1).1314 But it did not endorse two interventions in the measure—central venous pressure and blood oxygen saturation within six hours of a patient presenting with septic shock. In 2012 Henry Ford Hospital submitted an application for a new sepsis measure: SEP-1. Besides stipulating a larger fluid challenge and serial lactate measures, it differed little from the one originally submitted in 2008 and again required measurement of central venous pressure and blood oxygen saturation.15

Table 1

Timeline of events related to SEP-1

View this table:

In contrast to 2008, in March 2013 the National Quality Forum endorsed all of SEP-1 including the requirement for central venous pressure and blood oxygen saturation (table 1).1315 At the time, there was controversy regarding potential financial conflicts (see later) and the validity of the analysis of the seminal study supporting this requirement.161718 A body of published literature not supporting a physiological basis for central venous pressure and blood oxygen saturation in septic shock management19202122 led many sepsis experts to question these measures’ usefulness.2324252627 Two large studies had shown low physician compliance with the Surviving Sepsis Campaign guidelines, include these measures.24282930

Three multicentre randomised trials were started in 2008 to investigate the usefulness of measuring central venous pressure and blood oxygen saturation for sepsis and were nearing completion at the time of SEP-1’s endorsement.31 During the endorsement process eight professional societies raised concerns and asked the forum to wait for the results of the ongoing trials before endorsing SEP-1.13323334 Based on the forum’s 2013 endorsement, the Centers for Medicare and Medicaid Services announced that it would adopt all of SEP-1 for its hospital inpatient quality reporting programme, thereby requiring many doctors to report on their use of SEP-1’s interventions, whether or not they judged them best for their patients.35

Evidence provided in the SEP-1 application

The application submitted for SEP-1’s endorsement in 2012 included four sources of evidence to support the measurement of central venous pressure and blood oxygen saturation: the Surviving Sepsis Campaign guidelines, a retrospective analysis of 15 000 patients enrolled in the campaign, 55 referenced studies, and three meta-analyses.15 Each lacked scientific rigour and provided a weak basis for endorsing a performance measure (box 2).15303637 Other interventions in SEP-1, such as the need for a 30 mL/kg fluid challenge and serial lactate determinations, also failed to meet the National Quality Forum’s standards for scientific evidence.1536

Box 2

Critical analysis of evidence supporting central venous pressure and blood oxygen saturation

Surviving Sepsis Campaign guideline (2008)

Reported evidence for recommending use of these measures was graded in the guidelines as weak (grade C)

Retrospective analysis of 15 000 patients enrolled in the Surviving Sepsis Campaign

  • Data collection was not refereed

  • Analysis lacked a control group

  • Concluded no significant association between attaining targets and improved survival

Fifty five studies from the literature

  • 17 of the 55 studies provided no data comparing the survival effects of a sepsis bundle of performance measures to controls

  • 34 studies were observational studies

  • Four were randomised controlled trials; the seminal study supporting the measures was a single centre, non-blinded study criticised for its statistical analysis and the financial conflicts of its primary author. The other three were conducted in China where medical practice and patient mix differ from the West

Three meta-analyses

  • One concluded that the nine studies analysed were confounded and provided no convincing evidence supporting sepsis bundle use

  • The other two included studies that were among the 55 problematic ones noted above, that did not assess patients with sepsis, or that included protocols not comparable with the performance measure under evaluation

RETURN TO TEXT

Commercial interests

Edwards Lifesciences produces an oximetry catheter to continuously monitor central venous pressure and blood oxygen saturation and, therefore, stood to benefit from the measures being included in SEP-1. This company had paid or supported the research of the primary developer and two co-developers of SEP-1.30373839 AdvaMed is a medical technology trade association that promoted Edwards’s and other manufacturers’ devices.40 Edwards’s chief executive officer of has been the chair of AdvaMed’s committee on payment and healthcare delivery and the chair and a member of AdvaMed’s board of directors.4142 AdvaMed nominated its vice president of payment and healthcare delivery to the National Quality Forum’s steering committee reviewing SEP-1 (National Quality Form, personal communication, 2014), and this vice president was co-chair of the steering committee during SEP-1’s endorsement review.43 We know of no evidence that any person or company intended to or effectively influenced the forum’s 2013 decision to endorse SEP-1 and its requirement for central venous pressure and blood oxygen saturation. But the commercial relations are not in accordance with the forum’s policy on potential conflicts, indicate potential undue commercial influence, and weaken SEP-1’s credibility (box 1).

Reaching a compromise

In April 2014, the ProCESS trial of 1351 patients from 31 hospitals reported that protocol based haemodynamic therapy using central venous pressure and blood oxygen saturation did not significantly improve sepsis survival.31 After public comment, and with two more trials of the measures still under way, the Centers for Medicare and Medicaid Services announced in August 2014 that Henry Ford Hospital should collaborate with other stakeholders to reach a compromise on inclusion of the measures in SEP-1.44 The two other randomised controlled trials (ARISE and ProMISe) confirmed the absence of significant benefit for central venous pressure and blood oxygen saturation for routine septic shock management.31 Improving sepsis care over the past decade and selection bias have been postulated to explain why these three trials did not reproduce the findings of the original pivotal study.4546

The National Quality Forum subsequently endorsed changes to section F of SEP-1 based on the compromise reached between the measure developers and key stakeholders including the Society of Critical Care Medicine.47 Central venous pressure and blood oxygen saturation measures were still included.

The revised SEP-1 said that a volume status and tissue perfusion assessment could be fulfilled if all five components of a focused exam were completed by a licensed independent practitioner (capillary refill, cardiopulmonary examination, peripheral examination, skin examination, vital signs) or if two of four measures were taken (central venous pressure, blood oxygen saturation, bedside cardiovascular ultrasonography, or fluid responsiveness using leg raise or fluid challenge).48 A well intentioned provider might still consider central venous pressure and blood oxygen saturation as the best choices because the other options are either more expensive, infrequently used, or are not described or evaluated in standard adult sepsis guidelines. Continued inclusion of the measures in SEP-1 gives oximetry catheter manufacturers a powerful marketing tool to promote a non-beneficial device for routine sepsis management.49

The primary developer of SEP-1 continued to receive financial support from Edwards.38 Furthermore, two co-developers of SEP-1 with a past history of commercial relations with Edwards were both past presidents of the Society of Critical Care Medicine, an organisation with financial relationships with Edwards. One of the two SEP-1 co-developers was the society’s representative to the National Quality Forum.36505152 The Centers for Medicare and Medicaid Services notified hospitals that data collection for the revised SEP-1 would begin in fiscal year 2015.53

Conclusions

The Centers for Medicare and Medicaid Services adopting SEP-1 will help make an unproven commercial product a national standard of care for the management of sepsis. Trade associations or other commercial entities and their representatives with the potential to profit, even if indirectly, from a particular endorsement should be excluded from that endorsement process. At present, such entities can participate in and vote at every stage of the National Quality Forum’s endorsement process. To avoid apparent or unseen conflicts of interest, the US Food and Drug Administration excludes commercial representatives from the approval of new drugs or devices. Given the influence that performance measures endorsed by the National Quality Forum have on national healthcare, this agency should adopt similar practices. If governmental agencies do not ensure that performance measures are based on sound scientific evidence and are safeguarded from potential commercial influences and even the perception of conflicts of interest, trust in these agencies will be undermined and patient care compromised.

Acknowledgments

We thank Xizhong Cui, Amisha V Barochia, Junfeng Sun, Naomi O’Grady, and Anthony Suffredini for their valuable comments during the preparation of this manuscript. We also thank Kelly Byrne for expert editorial assistance.

Footnotes

  • Contributors and sources: All authors contributed to the planning, research, and preparation of the manuscript describing this work.

  • Competing interests: We have read and understood BMJ policy on declaration of interests and have none to declare. The opinions expressed in this manuscript belong to the authors and do not represent the National Institutes of Health or any other division of the US Department of Health and Human Services. The research was funded by the Division of Intramural Research of the Clinical Center at the National Institutes of Health.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

References

View Abstract

Log in

Log in through your institution

Subscribe

* For online subscription