The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines
BMJ 2016; 352 doi: https://doi.org/10.1136/bmj.i1152 (Published 08 March 2016) Cite this as: BMJ 2016;352:i1152
All rapid responses
Dear editor
There have been at least two systematic reviews of national and international clinical practice guidelines using the AGREE tools (I & II).1,2,3, Both found that no guideline was perfect and highlighted the same key problems – a lack of applicability and stakeholder involvement. These domains relate whether i) existing barriers and facilitators that impact on the application of a guideline have been considered and ii) whether the relevant professions had been involved in the guideline development, whether the view and preferences of the target population had been sought and whether the target users of the guideline are clearly defined.
These findings are important as both studies demonstrated that this has consequences for the utility of such guidelines in clinical practice. For example, the assessment of all current national and international chronic heart failure guidelines demonstrated considerable heterogeneity in the evidence presented in relation to end-of-life care.2 Given that all guidelines in this study had access to a similar (if the not the same) evidence base, such a degree heterogeneity was surprising. Whilst this could simply be a result of difference in the methods adopted in the development of guidelines, it also raises important questions about the social and organizational processes by which evidence become transformed into clinical guidelines. In light of this, tools such as the AGREE II undoubtedly have an important role to play in assessing the quality of clinical practice guideline development.3
References
1. Hegarty K, Gunn J, Blashki G, Griffiths F, Dowell T, Kendrick T., How could depression guidelines be made more relevant and applicable to primary care? A quantitative and qualitative review of national guidelines. Br J Gen Pract 2009;59:e149-56.
2. Irving G, Edwards J, Holden J, Reeve J, Dowrick C, Lloyd-Williams M. Chronic Heart failure guidelines: Do they adequately address patient need at the end-of-life? International Journal of Cardiology 2013.
3. Brouwers M, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, Fervers B, Graham ID, Grimshaw J, Hanna S, Littlejohns P, Makarski J, Zitzelsberger L for the AGREE Next Steps Consortium. AGREE II: Advancing guideline development, reporting and evaluation in healthcare. Can Med Assoc J. 2010. Available online July 5, 2010. doi:10.1503/cmaj.090449
Competing interests: No competing interests
Re: The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines
As recently pointed out by Djulbegovic, Bennett, and Guyatt [1, 2], one fundamental reason for the limited progress in improving delivery of health care is that quality improvement (QI) has been treated more as a top‐down, compliance, and regulatory issue rather than a matter deeply rooted in the science of decision making, evidence‐based medicine (EBM), and patient‐oriented care [1-3].
Let us take, as an example, the AGREE (Appraisal of Guidelines, Research, and Evaluation in Europe) initiative, which was started in 2000 [4]. This group published the first version of their critical appraisal check-list in 2003 [5]. This first publication met with a considerable success, eg there are more than 1300 citations of this publication in google scholar [5]. The second version was published in 2010 [6] and met with an even more considerable success ie more than 1900 citations in google scholar [6]. Finally, this group of thinkers published an improved version of their checklist in 2016 in the BMJ [7]. Hundreds, if not thousands of health-care professionnals worldwide have now used the AGREE, or AGREE-II, check-lists, and instruments, to evaluate the methodological quality of thousands of guidelines, leading to hundreds of publications, that have themselves been reviewed [8-10].
The main conclusion of these evaluations is that the methodologic quality of guidelines is rarely excellent, and that this situation tended to improve during the last decade [8-10]. It is quite possible that that the AGREE iniatiative facilitated this improvement, and it is therefore certainly a good starting point for health-care professionals to evaluate guideline quality. But this QI initiative is not enough: AGREE focuses on quality of guideline development, whereas it has been shown that methodologic quality is not sufficient to ensure that recommendations are appropriate/accurate ie in keeping with the principles of EBM [11-19]. Therefore health care professionals should also evaluate guideline content before they decide to implement any recommandation in daily practice.
The problem is that among the thousands of guidelines evaluations with the help of the AGREE check-list, and/or other similar check-lists, that have been published so far, only a few also evaluated guideline content [11-19]. And the latter evaluations often failed to demonstrate an obvious relation between quality and content. Therefore, using the very words of Djulbegovic, Bennett, and Guyatt [1, 2], maybe we could not affirm that AGREE has been harmful, but if AGREE were not able to improve in the future, then it might well end up becoming wasteful. Maybe it would be time now to think about AGREE-III ?
References:
[1] Djulbegovic B, Bennett CL, Guyatt G. Failure to place evidence at the centre of quality improvement remains a major barrier for advances in quality improvement. J Eval Clin Pract. 2019 Jun;25(3):369-372. Doi: 10.1111/jep.13146.
[2] Djulbegovic B, Bennett CL, Guyatt G. A unifying framework for improving health care. J Eval Clin Pract. 2019 Jun;25(3):358-362. doi: 10.1111/jep.13066.
[3] Born KB, Levinson W. Choosing Wisely campaigns globally: A shared approach to tackling the problem of overuse in healthcare. J Gen Fam Med. 2018 Dec 21;20(1):9-12. http://www.choosingwisely.org/wp-content/uploads/2015/04/About-Choosing-...
[4] Appraisal of Guidelines, Research, and Evaluation in Europe (AGREE) Collaborative Group. Guideline development in Europe. An international comparison. Int J Technol Assess Health Care. 2000 Autumn;16(4):1039-49.
[5] AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003 Feb;12(1):18-23.
[6] Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, Fervers B, Graham ID, Grimshaw J, Hanna SE, Littlejohns P, Makarski J, Zitzelsberger L; AGREE Next Steps Consortium. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010 Dec 14;182(18):E839-42. doi: 10.1503/cmaj.090449. http://www.cmaj.ca/content/182/18/E839
[7] Brouwers MC, Kerkvliet K, Spithoff K; AGREE Next Steps Consortium. The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines. BMJ. 2016 Mar 8;352:i1152. doi: 10.1136/bmj.i1152. Erratum in: BMJ. 2016 Sep 06;354:i4852.
[8] Alonso-Coello P, Irfan A, Solà I, Gich I, Delgado-Noguera M, Rigau D, Tort S, Bonfill X, Burgers J, Schunemann H. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care. 2010 Dec;19(6):e58. doi: 10.1136/qshc.2010.042077.
[9] Armstrong JJ, Goldfarb AM, Instrum RS, MacDermid JC. Improvement evident but still necessary in clinical practice guideline quality: a systematic review. J Clin Epidemiol. 2017 Jan;81:13-21. doi: 10.1016/j.jclinepi.2016.08.005.
[10] Gagliardi AR, Brouwers MC. Do guidelines offer implementation advice to target users? A systematic review of guideline applicability. BMJ Open. 2015 Feb 18;5(2):e007047. doi: 10.1136/bmjopen-2014-007047.
[11] Burgers JS. Guideline quality and guideline content: are they related? Clin Chem. 2006 Jan;52(1):3-4.
[12] Watine J, Friedberg B, Nagy E, Onody R, Oosterhuis W, Bunting PS, Charet JC, Horvath AR. Conflict between guideline methodologic quality and recommendation validity: a potential problem for practitioners. Clin Chem. 2006 Jan;52(1):65-72.
[13] Watine JC, Bunting PS. Mass colorectal cancer screening: methodological quality of practice guidelines is not related to their content validity. Clin Biochem. 2008 May;41(7-8):459-66. doi: 10.1016/j.clinbiochem.2007.12.020.
[14] Matthys J, De Meyere M. Quality evidence important for quality guidelines. CMAJ. 2010 Sep 21;182(13):1449-50. doi: 10.1503/cmaj.110-2105.
[15] Matthys J, De Meyere M, van Driel ML, et al. Differences among international pharyngitis guidelines : not just academic. Ann Fam Med. 2007;5:436–43.
[16] Nuckols TK, Lim YW, Wynn BO, Mattke S, MacLean CH, Harber P, Brook RH, Wallace P, Garland RH, Asch S. Rigorous development does not ensure that guidelines are acceptable to a panel of knowledgeable providers. J Gen Intern Med. 2008 Jan;23(1):37-44.
[17] Eikermann M, Holzmann N, Siering U, Rüther A. Tools for assessing the content of guidelines are needed to enable their effective use--a systematic comparison. BMC Res Notes. 2014 Nov 26;7:853. doi: 10.1186/1756-0500-7-853.
[18] Coates D, Homer C, Wilson A, Deady L, Mason E, Foureur M, Henry A. Induction of labour indications and timing: A systematic analysis of clinical guidelines. Women Birth. 2019 Jul 5. pii: S1871-5192(19)30141-6. doi: 10.1016/j.wombi.2019.06.004.
[19] Pallari E, Fox AW, Lewison G. Differential research impact in cancer practice guidelines' evidence base: lessons from ESMO, NICE and SIGN. ESMO Open. 2018 Jan 6;3(1):e000258. doi: 10.1136/esmoopen-2017-000258.
Competing interests: No competing interests