Ensuring the integrity of clinical practice guidelines: a tool for protecting patientsBMJ 2013; 347 doi: https://doi.org/10.1136/bmj.f5535 (Published 17 September 2013) Cite this as: BMJ 2013;347:f5535
- Jeanne Lenzer, medical investigative journalist1,
- Jerome R Hoffman, professor of medicine emeritus2,
- Curt D Furberg, professor of public health sciences emeritus3,
- John P A Ioannidis, professor of medicine4
- On behalf of the Guideline Panel Review working group
- 1NY, USA
- 2UCLA School of Medicine, CA, USA
- 3Wake Forest School of Medicine, NC, USA
- 4Stanford University, CA, USA
- Correspondence: Jeanne Lenzer:
Clinical practice guidelines should support doctors by identifying and disseminating the most scientifically sound healthcare practices. When performed rigorosly, this endeavor improves patient care and elevates the profession toward its scientific ideal. However, widespread financial conflicts of interest among the authors and sponsors of clinical practice guidelines have turned many guidelines into marketing tools of industry. Financial conflicts are pervasive, under-reported, influential in marketing, and uncurbed over time.1 2 3 4 5 6 7 Biased guidelines can cause grave harms to patients, while creating a dilemma for doctors, who may face professional or legal consequences when they choose not to follow guidelines they distrust.8 9 Such guidelines fail to place patients’ needs foremost, and instead protect livelihoods and preserve ideologies.
The Institute of Medicine and others10 11 12 13 have recommended that guideline authors should reduce or eliminate financial and professional conflicts. Unfortunately, these admonitions have had little effect.6 Until changes are made in the way panels are constituted, the best defense is an informed readership. Therefore, we address guidelines users and the journals that publish them. The biasing impact of financial conflicts—such as “panel stacking,” which we discuss below—can be invisible to guideline readers. For this reason we have created the Guideline Panel Review (GPR), a short list of questions for guideline authors, which we believe, if published with guidelines, will allow physicians, administrators, policy makers, and patients to better determine which guidelines are likely to offer reliable advice—and which are not.
The special nature of financial conflicts
It has been argued it is unfair to single out financial conflicts of interest because other forms of bias are common.14 We believe, however, the impact of financial conflicts is different from that of other forms of bias in two fundamental ways. Personal biases are generally, though not always, multidirectional; while some experts might prefer one approach to patient care for personal or philosophical reasons, others are likely to take a contrary stand. On the other hand, the bias introduced by financial conflicts of interest is almost invariably unidirectional for two key reasons. Firstly, almost all research is sponsored by industry, and as a result, negative outcomes tend to remain unpublished while positive conclusions are widely disseminated and promoted.15 16 17 18 Secondly, industry funds vast networks of lobbyists, patient groups, researchers, lawyers, medical writers, advertising and social networking specialists, and others, all of whom promote their products and counter their critics. These networks, backed by resources available to industry but rarely (if ever) to individual academics or community doctors, make financial conflicts of interest a powerful form of bias associated with vital breaches of public trust. Failure to take measures to reduce or eliminate this form of bias on grounds that other biases exist allows the pursuit of perfection to become the enemy of the good.
We are concerned with two major types of financial conflicts, the most obvious of which arises when a panelist or the sponsoring organization derives material benefit—such as consulting or speaker’s fees, research grant funding, stock ownership, or donations—from a commercial entity that stands to win or lose revenue on the basis of the guideline recommendation(s). The other type of financial conflict is a professional conflict, which arises when guideline creators are clinicians who specialize in the area under review.19 A notable example is self referral bias, in which doctors, after acquiring ownership interest in specialty hospitals, increased their referrals for expensive and invasive procedures twofold to 10-fold.20 Although professional conflicts might represent intellectual conflicts of interest arising through training or shared ideology, the distinction between intellectual and financial conflicts is blurred by the presence of the potential for financial gain. Psychology research suggests these types of financial self interest bias decision making in ways that are invisible to the decider, and thus are not easy for the individual involved to avoid, or for observers to detect.21
Financially conflicted panel leadership can extend the impact of bias even to those who may not have direct financial ties to industry through a process of “committee stacking,” or the selection of panelists known to support a desired outcome. This is especially worrying as a recent survey found 71% of clinical policy committee chairs and 90.5% of committee co-chairs had financial conflicts6 and the committee leadership often selects or approves committee members, thereby undermining the goal of objective scientific inquiry.
Other groups have recognized the importance of this problem, notably the Institute of Medicine, which wrote in-depth recommendations regarding conflict of interest in guidelines,10 and the Appraisal of Guidelines for Research and Evaluation (AGREE) consortium, which has developed multiple checklists to score guidelines (AGREE II, Global Rating Scale).12 13 22
Current guideline limitations
Although many organizations and individuals have warned about the biasing influence of financial conflicts of interest on guideline panels,2 4 10 11 21 23 24 25 26 27 28 29 30 some critics have opposed attempts to rein in financial conflicts saying guidelines should be judged solely on the integrity of the scientific review, and not on the reviewers of that science.31 However appealing this argument might be, it ignores empirical evidence that financial conflicts affect how reviewers select and analyze data, which can lead panels with conflicts to produce guidelines with starkly different recommendations from panels without financial conflicts.32 Furthermore, guidelines are generally produced and published without the tempering effect of peer review,8 leaving guideline users, who typically lack the time and expertise to review the science on its merits, to trust implicitly the guideline development process. This leap of faith is thus entirely dependent upon the integrity of that process, and the individuals and groups controlling it. Given these concerns, the method by which guidelines are created assumes paramount importance.
Guidelines can have a powerful effect on the behavior of clinicians. Highly publicized guidelines from prestigious institutions might be issued (and viewed) as clinical “rules,” making some doctors reluctant to deviate from recommendations, especially in the face of professional censure or potential legal consequences for failure to adhere to a “standard of care.”8 9 As a result, some doctors may practice “cookbook medicine,” indiscriminately following the rules rather than making nuanced decisions for patients. Nonetheless, one of the most important benefits of evidence based guidelines is the reduction of harmful or costly unjustified variation in patient care. Biased or non-evidence based guidelines can also reduce variation, even when they should not, thus potentially causing harm because the recommended intervention is suboptimal, ineffective, dangerous, or recommended for inappropriate patients.
Poor quality guidelines may also result when the available evidence is inadequate or conflicting, but guideline authors—who often document all the reasons the study results are unreliable—nonetheless combine the conflicted results to promote a single approach describing what clinicians should do, rather than acknowledging definitive recommendations would be inappropriate, and credible alternative approaches can be justified.
Content experts not only dominate most guideline panels, but also take authorship and dictate the conduct, conclusions, and interpretation of systematic reviews, which in turn provide the rationale for the subsequent guideline recommendations. Content experts or topic specialists are especially likely to have a financial or professional conflict of interest or both, increasing the risk of bias.4 24 27
Recommendations to assess bias in the constitution of guideline panels
The Institute of Medicine suggests bluntly that individuals with a financial conflict of interest should be recused from joining panels, and clinicians with a professional conflict that cannot be divested—for example, a heart surgeon assessing heart surgery—should constitute at most a minority of panelists.10 In line with these recommendations, we believe readers should be skeptical of guidelines if any panelists have a financial conflict, recognizing that fully meeting this high standard may not be easy. Nevertheless, there is a large pool of non-conflicted experts, even among subspecialty groups, who can fill guideline panels.33 34 Although the precise acceptable proportion of panel members with a professional conflict can be debated, business as usual is no longer an option.
We believe guideline panels must primarily comprise experts at reviewing scientific evidence.35 In current practice, such scientists are either not involved at all in guideline creation, or work in an advisory or consultant capacity to content experts. These roles should be inverted. The first task of any guideline panel is to review the evidence to decide if there is an uncontested “best” answer; if not, and scientifically based controversy exists, then the panel constituency must reflect that diversity of thought. “Panel stacking” with individuals known to believe disproportionately in one school of thought, must be avoided. Panels whose membership fairly represents differing scientific views are highly desirable. To assure guideline readers that panels are not stacked, panelists should declare their a priori beliefs about a proposed intervention at the time they are nominated, and again at the conclusion of the panel’s term. Many such declarations can be vetted because panelists often have published their viewpoints before empanelment.
Because content experts are generally conflicted when reviewing topics in their own specialty, they should be consulted for content/topical issues, but should not be the authors of the systematic reviews informing recommendations35 nor of the guidelines themselves. If content experts with a professional conflict are involved as authors of systematic reviews or guidelines, their inclusion should be explicitly justified.35 When a topic is controversial, content experts on both sides of any question may be invited to explain why they believe their approach is best.36
Guideline panels should include other stakeholders, such as patients or patient representatives and public interest or public health groups. It is also highly desirable for draft guidelines to be subjected to external review by independent entities and (when appropriate) public stakeholders.
A properly constituted guideline panel might find the evidence, even when meticulously reviewed, does not provide a clear conclusion. This is especially likely when data are withheld, unpublished, inadequate, or conflicting.37 If the evidence does not support straightforward conclusions, pretending it does is worse than admitting uncertainty. Guideline authors should decide how much uncertainty exists for each option under consideration, acknowledge the limits of the evidence, and present alternate opinions, with an emphasis on defining which patient and societal values are best served by each approach, and on providing information to support shared decision making.10 13 The greater the uncertainty, the more a guideline should exhibit restraint in recommending any one approach over another.
When issuing guidelines, panelists should not present their recommendations as “rules,” from which a deviation in practice is implied to be substandard. Even in the relatively unusual instance where research evidence strongly suggests a single best approach, there must be room for variation in practice, based on special patient characteristics, and shared decision making.
Finally, even when panelists generally agree about the weight of the evidence, guidelines should include comment about the degree to which such evidence is less than definitive, and solicit (and publish) a “minority report” from panelists with scientifically based dissenting positions.
Summary of “red flags” regarding the constitution of guideline panels
Although there is no evidence base on which we can ground precise recommendations about how guideline panels should be constituted, there are data showing that certain practices do create important biases. These biases in turn produce recommendations not supported by evidence, which have the potential to cause substantial harm if widely adopted. The accompanying Guideline Panel Review (GPR) in box 1 shows several circumstances that should be seen by readers as “red flags,” each of which should raise skepticism about the entire content of any guideline created in such a manner. We have organized this to reflect what we believe to be the relative importance of each of these elements, although we acknowledge that this is to some degree subjective, and welcome further thoughts and input from readers.
Because individual clinicians have neither the time nor resources to identify each of these problems independently, we believe journals should be required to publish the results of the GPR prominently at the front of any guideline it publishes. At the same time, given strong evidence that “disclosure” is not adequate to address bias,21 we recommend that journals decline to print any guideline if there is evidence of sufficient bias in the manner by which it was sponsored and/or its panel was constituted so as to make its entire contents highly suspect.
We emphasize that the GPR is a framework for future development, and we welcome comments and alternate suggestions about how precisely to operationalize and validate this rating scheme.
Box 1: Red flags that should raise substantial skepticism among guideline readers (and medical journals)
Sponsor(s) is a professional society that receives substantial industry funding;
Sponsor is a proprietary company, or is undeclared or hidden
Committee chair(s) have any financial conflict*
Multiple panel members have any financial conflict*
Any suggestion of committee stacking that would pre-ordain a recommendation regarding a controversial topic
No or limited involvement of an expert in methodology in the evaluation of evidence
No external review
No inclusion of non-physician experts/patient representative/community stakeholders
*Includes a panelist with either or both a financial relationship with a proprietary healthcare company and/or whose clinical practice/specialty depends on tests or interventions covered by the guideline
How current guidelines stack up
We have appraised the reliability of several guideline panels according to the proposed GPR and find that their recommendations might be biased, largely because of pervasive conflicts of interest.
How the Guideline Panel Reviews (GPRs) were conducted
Six reviewers independently evaluated selected guidelines, reviewing the guideline publication, and if available, supporting publications or websites providing information about the guideline development process, websites from the sponsoring organization, and/or information obtained through interviews or correspondence with the guideline chair or primary guideline author. Each guideline was then independently reviewed by an additional rater. When we were unable to directly vet an item from independent sources, we relied on the statements of the guideline authorizations themselves; in doing so, we chose to err on the side of caution, perhaps underestimating conflicts of interest since studies show that authors do not always declare their conflicts.38 39
We used the following assessments:
Red flag indicates an element known to introduce potential bias—for example, industry funding of authors.
Caution indicates an item we believe is an important part of guideline development, but for which there is not proof that bias is introduced by the presence of that element.
U is for uncertain, indicating the raters could not confidently score the element with the available information.
The following reviews were conducted:
Screening for prostate cancer with prostate specific antigen
Treatment of high blood cholesterol in adults
Alteplase (tPA) for the treatment of acute stroke
Treatment of depression in adults
The guideline panel that received our highest rating was the United States Preventive Services Task Force for its evaluation of routine prostate specific antigen-based screening for prostate cancer. The task force found harms with no evidence of net benefit. That was in contrast with the highly conflicted panel of the American Urological Association, which issued a “Best Practice” statement in 2009 recommending routine screening. In 2013, the association revised its recommendations using an improved methodology and panel development process, and issued a more conservative recommendation for screening.⇓
The panels issuing the 2001 and 2004 guideline and update for the National Cholesterol Education Project of the National Heart, Lung and Blood Institute—which greatly expanded the number of people for whom cholesterol lowering drugs were recommended—were profoundly compromised by financial conflicts.⇓ Guidelines promoting alteplase for stroke earned red flags in most categories, including widespread financial conflicts and panel stacking.⇓ Three guidelines for the treatment of mild to moderate depression were of particular interest because recommendations to use drugs and/or electroconvulsive therapy increased as red flags and financial conflicts increased.⇓ Finally, the panel recommending percutaneous coronary intervention was compromised by financial and professional conflicts, and excessive use of percutaneous coronary interventions has been documented.⇓ 45
Clinical guidelines affect millions of individuals, and the finding that the panels issuing these guidelines might be constituted in ways that circumvent full scientific debate is troubling. We hope that medical journals will insist that guideline panels respond to the eight items of the GPR (see box 1). We are especially concerned with the role of panel stacking, since it is possible to have a panel virtually free of financial conflicts that is completely biased; this can occur when committee chairs, who overwhelmingly have financial conflicts, select panelists known to support a singular (and often industry friendly) viewpoint to the exclusion of skeptical experts. We recommend that journals publish the GPR, and ideally that they decline to publish guidelines seriously compromised by signs of important bias. We believe this will not only help readers, but will also, over time, encourage guideline sponsors to do a far better job of choosing panels freer of conflict of interest.
Cite this as: BMJ 2013;347:f5535
Participants: JL conceived the central idea for the article as part two of her feature “Why we can’t trust clinical guidelines.”8 She selected the authors and working group members based on their known concerns regarding bias and guidelines. Using a modified Delphi process via email, the group identified key sources of bias among guideline panels, which was followed by a conference call. JRH wrote a first draft; JPAI, CDF, and JL added subsequent sections. The draft was circulated to the working group for review, revision, and ultimately approval by all members. MN, KK and JL conducted fact checking. JL is the guarantor.
Guideline Panel Review working group: Shannon Brownlee, senior vice president, Lown Institute; Richelle Cooper, associate professor of emergency medicine, UCLA; Daniel Fatovich, professor of emergency medicine, University of Western Australia; Kevin Klauer, assistant clinical professor, Michigan State University College of Osteopathic Medicine; Maryann Napoli, associate director, Center for Medical Consumers; David Newman, associate professor of emergency medicine, Mount Sinai School of Medicine; Ryan Patrick Radecki, assistant professor of emergency medicine, The University of Texas Health Science Center at Houston; Rita Redberg, professor of medicine, Division of Cardiology, UCSF School of Medicine; Vikas Saini, president, Lown Institute; David Schriger, professor of emergency medicine, UCLA; Robert Solomon, core faculty, emergency medicine residency, Allegheny General Hospital, Pittsburgh.
External guideline reviewers: John Abramson, lecturer, healthcare policy, Harvard Medical School; Lisa Cosgrove, associate professor, University of Massachusetts-Boston and network fellow, Edmond J Safra Center for Ethics Harvard University; Allen F Shaughnessy, professor of family medicine, Tufts University School of Medicine
Competing interests: We have read and understood the BMJ Group policy on declaration of interests and declare KK received industry funding in the past; he has terminated his ties to industry. RFR serves as a consultant to several insurance companies. VS was cofounder of a medical device company.
Provenance and peer review: Commissioned; not externally peer reviewed.