Endorsing performance measures is a matter of trust
BMJ 2018; 360 doi: https://doi.org/10.1136/bmj.k703 (Published 23 February 2018) Cite this as: BMJ 2018;360:k703
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Responses to the Comments of the Leaders of the National Quality Forum:
We do not state or imply in our report that the co-chairman on the NQF committee reviewing SEP-1 who was vice-president of payment and healthcare delivery at AdvaMed had a conflict of interest during his work with the NQF committee regarding SEP-1. In fact, we state clearly that we know of none and only that there is the appearance of one as follows: AdvaMed is a medical technology trade association that promotes the devices of its member companies including Edwards Lifesciences, a maker of catheters that measure central venous pressure (CVP) and central venous oxygen saturation (ScvO2). Edwards Life Sciences would potentially profit from a CMS adopted performance measure advocating the use of CVP and ScvO2 measurements during the care of septic patients. Edwards Life Science’s chief executive officer was the chair of AdvaMed’s committee on payment and healthcare delivery and the chair and a member of AdvaMed’s board of director either before or during SEP-1’s review by the NQF. Any potential conflicts of interest these relationships posed for the co-chairman may have been disclosed to the NQF during its vetting of committee members. We do not know whether this co-chairman was recused from participating in discussion on SEP-1. However, he did participate in that discussion and voiced to the committee his support of NQF’s adoption of SEP-1, which included the controversial requirement for CVP and ScVO2 measurements. This participation is evidenced by text from the 12/5/12 NQF Infectious Disease Committee’s review of SEP-1 provided below (1).
Steering Committee co-chairman on SEP-1 and AdvaMed vice president: “... I agree, there’s been a lot of thoughtful discussion. And I do agree with you (other co-chair), that you know, I haven’t heard anything compelling that would prevent moving forward (on SEP-1), so I’m in agreement with that. And over the NQF criteria is not – is not the same as academic, you know, ad criteria, so it’s a little different. If you look at the criteria for NQF, I’m comfortable that the developers met the standard. Any other comments on this?”
According to the NQF leaders, SEP-1 most recently underwent maintenance of endorsement review in 2016. We hope that as the NQF leaders indicate the CVP and ScVO2 measurements shown to have no benefit for septic patients and only increase costs will be finally removed from SEP-1. However, measurement of the CVP and ScVO2 were included in the volume status and tissue perfusion assessment in the versions of SEP-1 the Centers for Medicare and Medicaid (CMS) continued to mandate through the end of 2017(Version 5.2) (2, 3). They are still described and algorithms for their completion are also still included as part of this assessment in the specification manual for the version of SEP-1 effective though June 2018 (pages 22, 23, 45, 46 and 47 of Version 5.3) (3). The version of SEP-1 that becomes effective in July 2018 (Version 5.4) still requires a volume status and tissue perfusion assessment but does not stipulate what that assessment should consist of (3). But the references for the specification manual for Version 5.4 which are provided in support of the components in the measure are unchanged from references in the earlier versions that mandated CVP and ScvO2 measurements.
Dharmvir S. Jaswal MD
Charles Natanson MD
Peter Q. Eichacker MD
References
1. NQF Infectious Diseases Committee transcript of minutes 12-5-12. https://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&Item... Accessed April 2, 2018
2. NQF-Endorsed Voluntary Consensus Standards for Hospital Care, Version 5.0a. https://www.nhfca.org/psf/resources/Updates1/SEP-1 Measure Information Form (MIF).pdf Accessed April 1, 2018
3. The Joint Commission. Specifications Manual for National Hospital Inpatient Quality Measures. Version 5.2; Version 5.3, Version 5.4. Pages 22, 23, 45, 46 and 47 of the Version 5.3a specifications manual. https://www.jointcommission.org/specifications_manual_for_national_hospi... Accessed April 1, 2018
Competing interests: No competing interests
TO THE EDITORS:
As leaders of the National Quality Forum (NQF), with clinical expertise in emergency medicine and acute care, we are deeply concerned with several inaccuracies included in “Endorsing performance measures is a matter of trust”. This article discussed the Centers for Medicare & Medicaid Services’ (CMS) Severe Sepsis and Septic Shock Early Management Bundle (SEP-1) measure. The authors mischaracterized NQF’s disclosure of interest process for seating expert committee members, and presented a misleading account of the measure’s review process at NQF. Most importantly, in 2016, NQF completed a review of the measure specifications based on updated evidence and the measure no longer includes the issues discussed in the paper.
The authors imply that the co-chair of the committee that reviewed SEP-1 in 2013 was conflicted. The co-chair was an employee of AdvaMed, a medical technology trade association, which promotes global policies to advance medical technology.
Jaswal et al state that, “We know of no evidence that any person or company intended to or effectively influenced the forum’s 2013 decision to endorse SEP-1 and its requirement for central venous pressure and blood oxygen saturation but the commercial relations are not in accordance with the forum’s policy on potential conflicts.” This is inaccurate. The co-chair was thoroughly vetted through NQF’s standard disclosure of interest process, a robust and transparent process that offers an opportunity for NQF member and public comment. Among several requirements, the policy explicitly requires all prospective members to disclose direct and indirect relationships with several entities including biomedical device manufacturing or sales companies, health care measure development organizations, medical or other health care profession societies, hospitals, clinics, health information technology manufacturing or sales companies, and health plans.
Committee members with conflicts are recused from participating in any discussion or vote on the measure(s) of interest. In accordance with policy, the co-chair of the committee that reviewed SEP-1 was found not to have a direct or indirect relationship with the third party, nor did the co-chair have direct involvement in developing SEP-1 or any other measures under endorsement consideration. At the time of the 2013 SEP-1 review, the co-chair was responsible for conveying health care delivery policy related information to AvaMed’s member organizations. The fact that a prospective or current committee member has a relevant interest does not mean that a conflict of interest exists or that the individual cannot serve on a committee.
The authors’ assessment of NQF’s evaluation of the evidence is also misleading and incomplete. The author’s contend that certain components of the SEP-1 bundle failed to meet NQF’s criteria for Importance to Measure and Report during the 2012 endorsement of maintenance review. (The SEP-1 bundle passed NQF’s Importance to Measure and Report criterion and was recommended for endorsement by the review committee with 79 percent approval.) The evidence was based on the 2008 Sepsis Survival Campaign , and – like other areas of improving and changing medical evidence – has since evolved.. It is troubling that the authors did not mention the most recent maintenance of endorsement review of SEP-1 in 2016 included a synthesis of the literature for several updated components in the bundle. The updates are based on the Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock 2016 and graded by the Sepsis and Septic Shock 2016 Guidelines. Monitoring central venous pressure and central venous oxygen saturation are no longer required or included in the new measure specifications as of this 2016 review of the literature and emerging evidence. This is an important, factual omission from the Jaswal et al paper.
The National Quality Forum (NQF) is an independent, not-for-profit, membership-based organization that brings together various public- and private-sector organizations to reach consensus on how to measure quality in healthcare to make it better, safer, and more affordable.
NQF is committed to achieving consensus and balancing interests through transparency and consistency and continuously evaluates and improves the endorsement process in response to the needs and concerns of our multi-stakeholder membership and the public. We are of course open to changes and improvements in our disclosure of interest or conflict of interest policies, but such changes must be based on fact, not false assertions or innuendo.
Sincerely,
Elisa Munthali, MPH
Shantanu Agrawal, MD, MPhil
Melissa Mariñelarena, RN, MPA, CPHQ
Jaswal SJ, Natanson C, Eichacker PQ. Endorsing Performance Measures is a Matter of Trust. BMJ. 2018 Feb 23;360:k703. doi: 10.1136/bmj.k703.
http://www.survivingsepsis.org/Guidelines/Pages/default.aspx
http://www.survivingsepsis.org/Guidelines/Pages/default.aspx
National Quality Forum. Committee Guidebook for the NQF Measure Endorsement Process. Washington, D.C.; 2017. http://www.qualityforum.org/Measuring_Performance/Consensus_Development_.... Last accessed March 2018.
Competing interests: No competing interests
Trusting the Sepsis Measure “Analysis”
As original developers (ST, ER, ML, RPD) and stewards (ST, ER) for the Centers for Medicare and Medicaid Services’ (CMS) Severe Sepsis and Septic Shock Early Management Bundle (SEP-1) measure and as a former policy advisor for CMS (LT), we provide the following corrections for Jaswal and colleagues’ multiple false and erroneous statements in “Endorsing performance measures is a matter of trust” (1).
SEP-1 is a measure in the Inpatient Quality Reporting (IQR) program, which only requires hospitals to report measure data to CMS; it is not a performance-based program (2). No link exists between a hospital’s performance on SEP-1 and payment. CMS publishes detailed information about the IQR program that the authors may not have reviewed (2, 3). The CMS web page cited by the authors pertains to the health insurance market and does not mention SEP-1. The references do not support the authors’ premise that CMS uses SEP-1 “to regulate US healthcare” by withholding reimbursement (4).
The authors confuse the IQR program (which houses the SEP-1 measure) with the CMS value-based purchasing (VBP) programs (5). These programs operate under different rules that are described publicly and updated annually through the government rulemaking process, including notice and comment periods (5).
The authors also claim that poor performance on SEP-1 jeopardizes hospital accreditation status. This claim is false. Neither reporting nor performance on SEP-1 affects a hospital’s accreditation status (6).
The authors assert that with the “compelling incentives” dispatched above, CMS has “mandate[d]” care approaches in SEP-1. This claim is false. SEP-1 is a protocol-based strategy to measure sepsis care rendered in hospitals and does not “mandate” that a hospital or clinician take any specific intervention. There is no punishment, financial or otherwise, for failing to complete any action in SEP-1.
The authors mislead readers to believe that reassessment of volume status and tissue perfusion necessitates the use of central venous pressure (CVP) and central venous oxygen saturation (ScvO2) monitoring. The current publicly available specification manual from 2017 (version 5.3a) clearly states that these variables are not required (6). As now specified, the reassessment requirement is satisfied if a clinician states that a reassessment was performed. There is no need to document the means by which the clinician completed the reassessment (7). We note the authors relied on a legacy specification manual (version 5.0a). Although now 3 years out-of-date, even then the measurement of CVP and ScvO2 was a matter of clinician choice with alternatives to satisfy the measure, among which was simply a focused physical exam.
Also of concern, the authors advance the fallacy that because the measure previously held out CVP and ScvO2 as variables that could satisfy the reassessment requirement, the developers (or perhaps NQF, or CMS) promoted the sale of Edwards Lifescience’s proprietary oximetry catheter. In fact, the two physiologic measurements can be obtained from a central line made by any manufacturer. If one were to endorse the authors’ reasoning which includes allegations of inappropriate intent and industry contamination, then SEP-1 corruptly supports many industries: pharmaceuticals (by recommending the use of antibiotics), manufacturers of intravenous tubing and crystalloid vendors (by recommending intravenous fluids), point-of-care manufacturers (by including lactate measurement), and perhaps support laboratory supply vendors too (by requiring blood cultures). The authors not only make a false allegation, but reach an illogical conclusion. The SEP-1 measure does not endorse any private entity and solely exists to promote sepsis quality care improvement.
Finally, these authors have falsely claimed that at least one developer “continued to receive financial support from Edwards Lifesciences,” and have attempted to disparage two others by referring to their past publicly disclosed affiliations with the firm. The claim of any continuing relationship is false. The authors’ citation to a web page without an associated publication date does not validate their claim. The standard by which conflicts of interests are adjudicated in medicine is disclosure. In this case that standard has been met and exceeded as none of the developers had relationship with Edwards in 2012 when the measure first included hemodynamic variables.
The authors, from the National Institutes of Health, are familiar with the need to resolve conflicts of interest, and they should be cognizant of the damage false and defamatory statements can do to respected institutions and individuals alike (8).
Because of the egregious nature of these multiple false or erroneous claims, we believe this publication should be retracted.
Respectfully,
Sean R. Townsend MD
Lemeneh Tefera MD MSc
Emanuel P. Rivers MD MPH
Mitchell M. Levy MD
R. Philip Dellinger, MD
1. Jaswal SJ, Natanson C, Eichacker PQ. Endorsing Performance Measures is a Matter of Trust. BMJ. 2018 Feb 23;360:k703. doi: 10.1136/bmj.k703.
2. Centers for Medicare & Medicaid Services. Hospital Inpatient Quality Reporting Program. 9 September 2017 Accessed at www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/... on 22 February 2018.
3. Social Security Act § 1886(b)(3)(B)(vii)-(viii); 42 CFR § 412.64(d)(2); 81 Fed. Reg. 56939 (August 22, 2016).
4. Centers for Medicare & Medicaid Services. Health Insurance Marketplaces. Accessed at https://www.cms.gov/CCIIO/Programs-and-Initiatives/Insurance-Programs/ on 28 February 2018.
5. Centers for Medicare & Medicaid Services. Hospital Inpatient Quality Reporting Program. 11 September 2017 Accessed at https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Inst... on 27 February 2018.
6. The Joint Commission. Specifications Manual for National Hospital Inpatient Quality Measures. Accessed at www.jointcommission.org/specifications_manual_for_national_hospital_inpa... on 22 February 2018.
7. QualityNet. Fact Sheets for Measure Changes. Accessed at https://www.qualitynet.org/dcs/ContentServer?c=Page&pagename=QnetPublic/... on 28 February 2018.
8. The Wall Street Journal. Blood-Study Author Didn’t Disclose Conflicts. 14 July 2008. Accessed at https://www.wsj.com/articles/SB121598988902149353 on 28 February 2018.
Competing interests: No competing interests
Re: Trusting the Sepsis Measure “Analysis” (Endorsing performance measures is a matter of trust)
Response to the Original Developers and to the Stewards of SEP-1
In the opening five paragraphs of their letter Dr. Townsend et al ascribe claims about SEP-1 found nowhere in our paper and characterize the SEP-1 bundle as a simple reporting system of CMS with no consequences. Mandating a government reporting system with no consequences for non-compliance defies credulity. To their first point that SEP-1 is not a performance-based measure, the Specifications Manual for National Hospital Inpatient Quality Measures for discharges from 10-01-15 through to the present states that SEP-1 is a Performance Measure and that its name is the Early Management Bundle, Severe Sepsis and Septic Shock (1, 2). Next we do not state anywhere in our article that a hospital’s payment is based on SEP-1 performance or that SEP-1 is part of the value based purchasing program (VBP). However, as a CMS core measure, SEP-1 is a candidate measure to be included in CMS’s VBP and this might be expected to happen in the near future since the measure is now part of the Hospital Compare program (3, 4, 5). Further, there will be potential costs to hospitals based on their reporting of SEP-1 data from 2017 as follows. CMS has announced that public reporting of SEP-1 in the Hospital Compare program will begin in July 2018 (4). This process will start with the compliance data hospitals reported to CMS for the first three quarters of 2017 and all compliance data for 2017 will be reported in October 2018. Through this Hospital Compare program, reporting data for hospitals non-compliant with this all-or-none measure during 2017 and later will be accessible by insurers and potential patients and referring physicians (5). Therefore, consumers will make decisions as to what hospitals their healthcare dollars are or are not spent at based in part on this publicly available SEP-1 compliance data. Hospitals may very well be penalized if during 2017 they either did not report their SEP-1 compliance data to CMS or if in reporting data it is evident that hospitals were noncompliant with this all-or-none performance measure. Another cost to hospitals of SEP-1 is that they have had to expend valuable resources to develop systems to record and document the extensive and complex data required to demonstrate to CMS compliance with SEP-1. Finally, we do not state anywhere that compliance with SEP-1 is presently necessary for hospital accreditation. However, as hospital payment becomes increasingly dependent on compliance with the SEP-1, the measure might also be expected to become a basis for hospital accreditation as Johns Hopkins medical center and others have noted and prepared for (6, 7).
We are not sure where the Stewards got their information about the contents of SEP-1. The facts about measurement of CVP and ScVO2 as published in the different versions of SEP-1are as follows. Measurement of central venous pressure (CVP) and central venous oxygen saturation (ScVO2) were included in the volume status and tissue perfusion assessment in the versions of SEP-1 the Centers for Medicare and Medicaid (CMS) continued to mandate through the end of 2017(Version 5.0 and 5.2) (1, 2). They are also still described and algorithms for their completion are still included as part of this assessment in the specification manual for the version of SEP-1 effective though June 2018 (pages 22, 23, 45, 46 and 47 of Version 5.3) (2). The version of SEP-1 that becomes effective in July 2018 (Version 5.4) still requires a volume status and tissue perfusion assessment but does not stipulate what that assessment should consist of. But the references for the specification manual for Version 5.4 which are provided to support application of the measure’s components are unchanged from references in earlier versions that mandated CVP and ScvO2 measurements, including the study by the developer of SEP-1 that originally promoted the use of early goal directed therapy and these measurements. While physicians may choose to use these measures in selected patients, we hope that reference or inferred use of these two measures which were shown reproducibly in randomized controlled trials to increase costs with no benefits to septic patients more than four years ago, will be completely removed from this government mandated performance measure in 2018.
We do not state that the NQF or CMS promoted the sale of Edwards Lifescience’s proprietary oximetry catheter. For the last two years, Edwards Life Sciences has used SEP-1 as a marketing tool for their oximetry catheter and other hemodynamic monitoring products (8). They state in these promotional materials that “Compliance with this measure (SEP-1) will determine payment starting fiscal year 2017”and “Edwards offers noninvasive and minimally invasive monitoring solutions with critical dynamic and flow based parameters to help you make proactive decisions, while also meeting the NQF #0500 (SEP-1) requirements”. These materials also present versions of SEP-1 that stipulate CVP and ScvO2 monitoring.
Edwards Life Science’s has provided on its web site promotional material for Early Goal Directed Therapy copyrighted in 2014 and as recently as 2018 stating that “Dr. Emmanuel Rivers is a paid consultant to Edwards Life Sciences” and that “Early Goal-Directed Therapy and EGDT are trademarks of Dr. Emmanuel Rivers”(9). An earlier version of these materials copyrighted 2014 during endorsement of SEP-1 no longer has an electronic link but can be obtained upon request.
Dharmvir S. Jaswal MD
Charles Natanson MD
Peter Q. Eichacker MD
References
1. NQF-Endorsed Voluntary Consensus Standards for Hospital Care, Version 5.0a. https://www.nhfca.org/psf/resources/Updates1/SEP-1 Measure Information Form (MIF).pdf Accessed April 1, 2018
2. The Joint Commission. Specifications Manual for National Hospital Inpatient Quality Measures. Version 5.2; Version 5.3, Version 5.4. Pages 22, 23, 45, 46 and 47 of the Version 5.3a specifications manual. https://www.jointcommission.org/specifications_manual_for_national_hospi... Accessed April 1, 2018
3. Hospital Value Based Purchasing Program. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Inst.... Accessed March 1, 2018
4. Hospital Inpatient Quality Reporting (IQR) Program. SEP-1 Early Management Bundle, Severe Sepsis/Septic Shock: v5.3a Measure Updates. Presentation transcript. February 27, 2018. https://www.qualityreportingcenter.com/event/sep-1-early-management-bund... Accessed April 1, 2018
5. CMS.gov, Centers for Medicare and Medicaid Services. Hospital Compare. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Inst... Accessed April 1, 2018
6. Department of Medicine, Clinical Affairs, Johns Hopkins Medicine. Medicine Matter. The sepsis core measure. https://medicine-matters.blogs.hopkinsmedicine.org/2016/03/the-sepsis-co... Accessed April 1, 2018
7. Faust JS and Weingart SD. The past, present and future of the Centers for Medicare and Medicaid Services Quality Measure SEP-1. Emerg Med Clinic N Am. 2017; 35: 219-231
8. Sepsis Facts. https://www.edwards.com/therapies/sepsis-management Accessed April 1, 2018
9. Edwards Lifesciences and Early Goal Directed Therapy 2018. https://www.edwards.com/eu/products/mininvasive/pages/egdtprotocol.aspx?... Accessed April 1, 2018
Competing interests: No competing interests