Quality of evidence is a key determinant for making a strong GRADE guidelines recommendation

doi:10.1016/j.jclinepi.2014.12.015

Journal of Clinical Epidemiology

Volume 68, Issue 7, July 2015, Pages 727-732

https://doi.org/10.1016/j.jclinepi.2014.12.015 Get rights and content

Abstract

Objectives

The objective of the study was to assess the association between Grading of Recommendations Assessment, Development and Evaluation (GRADE) factors and the strength of recommendations.

Study Design and Setting

The study was conducted as part of the development of clinical practice guideline (CPG) by American Association of Blood Banking related to role of prophylactic vs. therapeutic transfusion for the management of thrombocytopenia. The association between GRADE factors and strength of recommendations was assessed using logistic regression and multilevel mixed effect logistic regression model.

Results

Seventeen members of the CPG panel participated in the recommendation process. The quality of evidence was the only statistically significant (odds ratio = 4.5; P < 0.001) GRADE factor associated with the strength of recommendations. The predictive model showed that there is about 90% probability that panelists would issue the same (strong) recommendation when confidence in the effects of intervention is high vs. 10% when the quality of evidence is very low.

Conclusion

The results showed that quality of evidence is a key determinant for making a strong vs. a weak recommendation.

Introduction

What is new?

•
How factors like confidence in the estimates of health interventions' effects (quality of evidence), balance of benefit and harms, use of resources, and patients' value and preferences that the Grading of Recommendations Assessment, Development and Evaluation Working Group believes should impact on the strength of recommendations do in fact impact is not known
•
In this guideline development process, quality of evidence (confidence in intervention's effect) proved the key determinant of the strength of recommendations; at least for some guidelines, other factors did not appear to play a major role.

Development of a clinical practice guideline (CPG) is a complex process [1] that begins with formulation of specific clinical questions followed by systematic review of literature with the goal of identifying and critically appraising the relevant evidence, summarizing the evidence to address specific questions, and developing recommendation in favor or against a particular health intervention [2].

Using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group approach to grading recommendations, a strong recommendation implies uniformity of practice (just do it), whereas a weak recommendation implies that variation in practice is appropriate (action should be tailored to individual values and preferences through shared decision making) [3]. Although a systematic review of the relevant evidence addressing a clinical question is a key requirement, availability of evidence warranting high confidence (high-quality evidence) is not a guarantee of making strong recommendation for clinical practice. For instance, the desirable and undesirable consequences of an intervention may be closely balanced; values and preferences for outcomes may be very different across patients; or resource use (costs) may be significant. In any of these circumstances, one may have high confidence in estimates and, because fully informed patients may make different choices, a weak recommendation will be warranted.

Therefore, to formulate CPGs, the GRADE has recommended consideration of not only confidence in effect estimates of health intervention's effects, but also balance of benefits and harms, patients' values and preferences and consideration of resource use (GRADE factors) [4]. In an earlier empirical study, we showed that the higher the confidence in estimates, the greater probability that the guidelines panel will issue a strong recommendation [5]. However, that study had not incorporated other GRADE factors. The goal of this study was to assess the impact of all GRADE factors on the strength of recommendations.

Section snippets

Methods

The study was conducted as part of the CPG development process of American Association of Blood Banking (AABB) [6] related to use of prophylactic vs. therapeutic platelet transfusion in patients with thrombocytopenia. The AABB panel included 21 experts. Fifteen participants were hematologists or pathologists with expertise in transfusion medicine. Five additional panel members included a neurosurgeon, a cardiac surgeon, a critical care specialist, an anesthesiologist, and a hematologist. The

Results

Eighteen members of the panel participated in the deliberation of making recommendations and completed the online questionnaire. Fifty-five percent of the members were female (10 of 18). Seventeen percent (3 of 18) had experience of less than 5 years in their respective clinical area of practice, 28% (5 of 18) had 5 to 10 years of experience, and 55% (10 of 18) had experience of greater than 10 years. The summary results and distribution of responses for key questions are provided in Table 1,

Discussion

This is the first empirical study assessing the impact of all GRADE factors on the strength of recommendations. We found that, despite a strong inclination for panelists to rate factors in the same direction on the five-point scales (i.e., some panelists were inclined to rate confidence in estimates as high, benefits clearly outweighing harms, resources as not important, and patients wanting the intervention, and other panelists were inclined to rate all four factors in the opposite direct—see

References (14)

J. Andrews et al.
GRADE guidelines: 14. Going from evidence to recommendations: the significance and presentation of recommendations
J Clin Epidemiol
(2013)
J.C. Andrews et al.
GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation's direction and strength
J Clin Epidemiol
(2013)
G. Guyatt et al.
GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables
J Clin Epidemiol
(2011)
H. Balshem et al.
GRADE guidelines: 3. Rating the quality of evidence
J Clin Epidemiol
(2011)
G.H. Guyatt et al.
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
BMJ
(2008)
A. Qaseem et al.
The development of clinical practice guidelines and guidance statements of the American College of Physicians: summary of methods
Ann Intern Med
(2010)
B. Djulbegovic et al.
Impact of quality of evidence on the strength of recommendations: an empirical study
BMC Health Serv Res
(2009)

There are more references available in the full text version of this article.

Cited by (40)

Clinical recommendations: The role of mechanisms in the GRADE framework
2022, Studies in History and Philosophy of Science
Citation Excerpt :
As a recall, the strength of a GRADE recommendation is primarily based on: (1) the quality of evidence, (2) tradeoffs between desirable and undesirable effects, (3) variability in patient values and preferences, and (4) resource allocation (Guyatt et al., 2008c). Although the developers of GRADE do not claim that the strength of a recommendation is systematically tied to the quality of evidence (strong recommendations can occasionally be made on the basis of low-quality evidence (Andrews et al., 2013a)), Djulbegovic and colleagues have shown that, in practice, the certainty of evidence is the strongest determinant of the strength of a GRADE recommendation (Djulbegovic et al., 2015). At any rate, the literature on GRADE seems to suggest that using the framework should always result in making a clinical recommendation.7
The Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework has become one of the most influential frameworks for assessing quality of research and developing clinical recommendations. The GRADE framework has been presented as an evolution in the Evidence-Based Medicine (EBM) movement. Both GRADE and EBM emphasize effect estimates derived from population-level clinical trials and, as a consequence, devalue the role of mechanisms as the basis for clinical decisions. Although mechanisms do not hold the epistemic privilege of rigorous clinical trials in EBM reasoning, this paper will argue that mechanisms appear to be important in the use and application of GRADE, as described in the literature. The seemingly necessary role of mechanisms in the development of clinical recommendations has, so far, received little attention and is not explicitly featured in the literature describing GRADE. The analysis of the GRADE framework presented in this paper reveals an apparent tension between EBM's willingness to downplay mechanisms and what seems their inevitable use in GRADE. In this paper, we take the position that if mechanistic reasoning is inevitable in the use of GRADE, then the instructional literature on the framework would benefit from more explicit discussion of how to consider and integrate mechanisms.
Improving Guideline Adherence in Urology
2022, European Urology Focus
Citation Excerpt :
Clinical practice guidelines (CPGs) distil an evidence base into recommendations. These recommendations typically carry a strength rating (strong or weak) and an indication of the certainty of evidence (high, moderate, low, or very low) [1]. A judicious systematic review process based on transparent and reproducible methodology ideally underpins these recommendations [2].
Clinical practice guidelines (CPGs) distil an evidence base into recommendations. CPG adherence is associated with better patient outcomes. However, preparation and dissemination of CPGs are a costly task involving multiple skilled personnel. Furthermore, dissemination alone does not ensure CPG adherence. Reasons for nonadherence are often complex, but understanding practice variations and reasons for nonadherence is key to improving CPG adherence and harmonising clinically appropriate and cost-effective care.
To overview approaches to improving guideline adherence, to provide urology-specific examples of knowledge-practice gaps, and to highlight potential solutions informed by implementation science.
Three common approaches to implementation science (the Knowledge-To-Action framework, the Consolidated Framework for Implementation Research, and the Behaviour Change Wheel), are summarised.
Three implementation problems in urology are illustrated: underuse of single instillation of intravesical chemotherapy in non–muscle-invasive bladder cancer, overuse of androgen deprivation therapy in localised prostate cancer, and guideline-discordant imaging in prostate cancer. Research using implementation science approaches to address these implementation problems is discussed.
Urologists, patients, health care providers, funders, and other key stakeholders must commit to reliably capturing and reporting data on patient outcomes, practice variations, guideline adherence, and the impact of adherence on outcomes. Leverage of implementation science frameworks is a sound next step towards improving guideline adherence and the associated benefits of evidence-based care.
Clinical practice guideline documents are created by expert panels. These documents provide overviews of the evidence for the tests and treatments used in patient care. They also provide recommendations and it is expected that in most circumstances clinicians will follow these recommendations. Sometimes, health care professionals cannot or do not follow these recommendations and it is not always clear why. In this review article we look at some examples of research approaches to addressing this problem of nonadherence and we provide some examples specific to urology.
Communication is a two-way street: Analyzing practices undertaken to systematically transfer audit research knowledge to policymakers
2021, Accounting, Organizations and Society
Audit academics and policymakers express ongoing concerns about limited knowledge transfer between audit research and policymaking. We use theory-based knowledge transfer norms to evaluate eight practices used by audit academics to transfer research knowledge to policymakers. The discontinued PCAOB-AAA Auditing Section “Research Synthesis Project” came closest to enacting these theory-based knowledge transfer norms. Hence, we examine why those involved in that project did not follow these norms. Interviews with project authors reveal that their review creation approach anchored on the traditional academic literature review, and insufficiently adjusted it to meet the goal of communicating audit research evidence to policymakers. However, interviews with PCAOB project liaisons indicate that policymakers were engaged with and found value in the review creation process. Thus, we analyze PCAOB rulemaking documents for evidence that policymakers valued and used the project’s reviews in their policymaking. Over the project’s duration, we find increasing citations of research to the reviews themselves and to a broader set of academic research. Our findings of knowledge transfer occurrence in this project warrant further research on the efficacy of mobilizing audit research via research syntheses. We also show how several current audit domain knowledge transfer practices can be combined with the research syntheses approach leading to systematic effective knowledge transfer to policymakers.
Certainty of evidence and intervention's benefits and harms are key determinants of guidelines’ recommendations
2021, Journal of Clinical Epidemiology
Citation Excerpt :
Our earlier research demonstrated a strong interaction between CoE and BH_balance [13]. Both fundamental principles of EBM [7] and empirical data [13,27,28] support a relationship between CoE and SoR- the higher CoE, the closer findings will be to the “truth”, resulting in higher likelihood of endorsing such an intervention when benefits outweigh harms. Thus, without consideration of CoE and BH_balance, it is impossible to make evidence-based recommendations.
Many factors are postulated to affect guidelines developments. We set out to identify the key determinants.
a) Web-based survey of 12 panels of 153 “voting” members who issued 2941 recommendations; b) qualitative analysis of 13 panels of 311 attendees (panel members, systematic review teams and observers).
Compared with “no recommendations”, when intervention's benefit outweigh harms (BH-balance), probability of issuing strong recommendations in favor of intervention was 0.22 (95%CI: 0.08 to 0.36) when certainty of evidence (CoE) was very low; 0.5 (95%CI:0.36 to 0.63) when low; 0.74 (95%CI 0.61 to 0.87) when moderate and 0.85 (95%CI:0.71 to 1.00) when high. No other postulated factor significantly affected recommendations. The findings are consistent with a J- curve model when recommendations are issued in favor but not against an intervention. Panelists often changed their judgments as a result of the meeting discussion (67% for CoE to 92% for balance between benefits and harms). The panels spent over 50% of their time debating CoE; the chairs and co-chairs dominated discussion.
CoE and BH-balance are key determinants of recommendations in favor of an intervention. Chairs and co-chairs dominate discussion. Panelists often change their judgments as a result of panel deliberation.
Overall bias and sample sizes were unchanged in ICU trials over time: a meta-epidemiological study
2019, Journal of Clinical Epidemiology
Citation Excerpt :
Within the intensive care unit (ICU) setting, smaller trials have been shown to report larger effect estimates compared with larger trials, in part due to lower methodological quality [10]. Biased data and data from small trials may therefore result in unwarranted clinical practice recommendations of futile or even harmful healthcare interventions with widespread consequences for patients and society [11]. Over time, the risk of bias in RCTs has decreased both generally [12,13] and within specific fields [14].
To assess time trends in risk of bias (RoB) and sample sizes in randomized clinical trials (RCTs) of adult intensive care unit (ICU) patients.
A meta-epidemiological study of RCTs from Cochrane systematic reviews assessing interventions in adult ICU patients. Using run charts, we assessed time trends in the annual proportion of RCTs with overall low RoB, the annual median sample sizes, and the annual proportion of RCTs with low, unclear, and high RoB in individual bias domains.
We included 604 RCTs published between 1977 and 2018 from 53 Cochrane systematic reviews. Only 6.8% of the RCTs had overall low RoB. We observed only random variation in the annual proportions of RCTs with overall low RoB, in the annual median sample sizes and in most individual bias domains. For “allocation concealment,” we observed an increase in the proportion of low RoB RCTs and a decrease in the unclear RoB RCTs.
Few RCTs in adult ICU patients had overall low RoB. We found no evidence of an increase in RCTs with overall low RoB or in the median sample sizes over time. The only individual RoB domain with better ratings over time was “allocation concealment.”
Structured decision-making drives guidelines panels’ recommendations “for” but not “against” health interventions
2019, Journal of Clinical Epidemiology
Citation Excerpt :
For example, the importance of intolerance of uncertainty can be seen as a response to the underlying clinical uncertainties that activate analytical reasoning processes that prompted development of guidelines in the first place [35]. Our results regarding the importance of certainty of evidence are consistent with observations in two smaller studies [36,37]. The results provide empirical verification of the key EBM normative principle regarding the relationship between the credibility of underlying evidence and willingness to endorse a health intervention [1]: when certainty of evidence is high, we can expect that most panelists will issue strong recommendations.
The determinants of guideline panels’ recommendations remain uncertain. The objective of this study was to investigate factors considered by members of 8 panels convened by the American Society of Hematology (ASH) to develop guidelines using Grading of Recommendations Assessment, Development, and Evaluation (GRADE) system.
Web-based survey of the participants in the ASH guidelines panels. Analysis: two-level hierarchical, random-effect, multivariable regression analysis to explore the relation between GRADE and non-GRADE factors and strength of recommendations (SOR).
In the primary analysis, certainty in evidence [OR = 1.83; (95CI% 1.45–2.31)], balance of benefits and harms [OR = 1.49 (95CI% 1.30–1.69)] and variability in patients’ values and preferences [OR = 1.47 (95CI% 1.15–1.88)] proved the strongest predictors of SOR. In a secondary analysis, certainty of evidence was associated with a strong recommendation [OR = 3.60 (95% CI 2.16–6.00)] when panel members recommended “for” interventions but not when they made recommendations “against” interventions [OR = 0.98 (95%CI: 0.57–1.8)] consistent with “yes” bias. Agreement between individual members and the group in rating SOR varied (kappa ranged from −0.01 to 0.64).
GRADE's conceptual framework proved, in general, to be highly associated with SOR. Failure of certainty of evidence to be associated with SOR against an intervention, suggest the need for improvements in the process.

View all citing articles on Scopus

View full text

Original ArticleQuality of evidence is a key determinant for making a strong GRADE guidelines recommendation

Abstract

Objectives

Study Design and Setting

Results

Conclusion

Introduction

Section snippets

Methods

Results

Discussion

J Clin Epidemiol

J Clin Epidemiol

J Clin Epidemiol

J Clin Epidemiol

GRADE: an emerging consensus on rating quality of evidence and strength of recommendations

BMJ

The development of clinical practice guidelines and guidance statements of the American College of Physicians: summary of methods

Ann Intern Med

Impact of quality of evidence on the strength of recommendations: an empirical study

BMC Health Serv Res

Original Article
Quality of evidence is a key determinant for making a strong GRADE guidelines recommendation