Can we trust strong recommendations based on low quality evidence?BMJ 2021; 375 doi: https://doi.org/10.1136/bmj.n2833 (Published 25 November 2021) Cite this as: BMJ 2021;375:n2833
A necessary requirement for development of trustworthy guidelines is to respect the relation between the quality (certainty) of evidence and strength of recommendations. Strong recommendations are justified when they are based on high quality evidence, because such recommendations are considered more accurate.1 On the other hand, uncertainty in benefits and harms (that is, low quality evidence) generally leads to weaker recommendations.
The failure to recognise this important principle results in a tendency to issue strong recommendations based on low quality evidence (which we call discordant recommendations), often leading to harm. For instance, based on advice from low quality evidence, women have experienced avoidable adverse effects from hormone replacement therapy prescribed for the prevention of cardiovascular disease; and women with breast cancer have undergone highly toxic stem cell transplantation without benefit. This practice of decoupling the quality of evidence from the strength of recommendations is usually justified by separating guidelines into consensus guidelines versus evidence based guidelines—a practice that does not appear to have abated over time.
Basing treatment decisions or clinical guidelines on low quality evidence means that the true effects of a treatment or clinical decision might differ considerably from best estimates. This discrepancy could result in launching campaigns (such as those designed to persuade women to use hormone replacement therapy) that are based on an unjustified faith in net benefit instead of transparently sharing the uncertainties in the quality of evidence on which the recommendations were based. Inappropriately strong recommendations have other problematic consequences, such as discouraging future randomised controlled trials that would generate higher quality evidence.
However, not all discordant recommendations are equally problematic.2 For instance, patients with a high likelihood of bad outcomes might all be willing to try an unproven, but potentially beneficial intervention.
Issuing discordant recommendations without a compelling rationale is not unusual. The problem has previously been highlighted by the World Health Organisation3 and Endocrine Society,4 and most recently in our study in The BMJ.5 We found that when the American College of Cardiology (ACC)/American Heart Association (AHA) and the American Society of Clinical Oncology (ASCO) (the two largest worldwide organisations that develop guidelines for heart disease and cancer, the two leading causes of death globally) faced low quality evidence, 41% and 20% of their recommendations proved to be inappropriate or discordant, respectively.5 Inappropriate discordant recommendations are those that do not meet the GRADE (grading of recommendations assessment, development, and evaluation) criteria of appropriateness. Although these leading organisations claim to use evidence based methods for their guidelines, the fact that up to 41% of their recommendations are inappropriate and discordant should raise concerns in both health professionals and patients.
Some organisations—including the ACC/AHA and ASCO—explicitly classify their guidelines as evidence based when much of the supporting evidence is deemed to be moderate or high quality, and classify their guidelines as consensus based when it is not. In their consensus versus evidence based guidelines, the odds of issuing inappropriate discordant recommendations proved 2.6 times higher in ACC/AHA guidelines and 5.1 times higher in ASCO guidelines.5 Classifying guidelines as consensus based might allow panels to be less rigorous in ensuring that the strength of recommendations are consistent with the underlying quality of evidence. All guidelines require judicious consideration of the relevant evidence—in other words, all guidelines should be evidence based—and organisations should focus on avoiding inappropriate discordant recommendations.
When facing low or very low quality evidence, guidelines should avoid issuing inappropriate discordant recommendations. Abandoning consensus based guidelines is likely to facilitate this goal.
Competing interests: none declared.