We have read and used AMSTAR 2 with great interest [1]. This instrument has made a major revision based on the original AMSTAR and will be used to assess systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. However, when we use the AMSTAR 2 instrument to assess the quality of systematic reviews including both randomised and non-randomised studies, debates on some items have been highlighted.
Item 2 (a protocol prior to conduct), it’s extremely important to develop a protocol of systematic review in advance. AMSTAR 2 answers this item as “Yes”, “Partial Yes”, and “No”. However, only a small minority of published non-Cochrane reviews reported a protocol [2]. When a protocol of systematic review doesn’t exist, “Not applicable” or “No protocol available” should be considered.
Item 4 (Literature search strategy), AMSTAR 2 has considered all key contents except for the reliability of the literature search. Methodological studies and guidelines have highlighted the importance of searchers, involving a local healthcare librarian or information specialist could effectively avoid errors during the conduct of the search [3-5]. Therefore, it’s important to determine the reliability of search by identifying whether systematic reviewers involve an information specialist or whether the search strategies were peer-reviewed.
Item 7 (provide a list of excluded studies and justify the exclusions), this item should contain two parts, one is a list of excluded studies, and another is the reasons for exclusion. Our experience tell us most published non-Cochrane reviews don’t provide the list of excluded studies, but give a reason of exclusion and (or) a c. When systematic reviews just report the reason of exclusion and (or) a PRISMA flow graph, should evidence users answer this item as “Partial Yes” or “No”?
Item 11 (appropriate methods for statistical combination), AMSTAR 2 indicates that authors should report pooled estimates separately for the different study types. In fact, some systematic reviews including both randomised and non-randomised studies combine the estimates from two study types. And there are some methodological studies to support the combination of evidence from randomised and non-randomised studies [6,7]. However, it’s more important to justify the combination methods for different study types.
Item 15 (investigation of publication bias), quantitative synthesis is performed, authors should investigate the likelihood and magnitude of publication bias. “Yes” are answered when performing graphical or statistical tests for publication bias. However, the Cochrane Handbook doesn’t recommend performing graphical or statistical tests for publication bias when the number of included studies is less than 10. In this situation, how do we assess this item?
The original AMSTAR has been a popular instrument for critically assessing systematic reviews of randomised studies. We believe that the current updated version will be more popular and more operable for researchers to assess the quality of systematic reviews and for evidence users to identify high quality systematic reviews.
Competing interests: No competing interests
References
1. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017,358:j4008. doi: 10.1136/bmj.j4008.
2. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med 2007, 4, e78.
3. Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.
4. Li L, Tian J, Tian H, et al. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol. 2014, 67:1001-7. doi: 10.1016/j.jclinepi.2014.04.003.
5. Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol 2009;62: 944e52. doi: 10.1016/j.jclinepi.2008.10.012.
6. Verde PE, Ohmann C. Combining randomized and non-randomized evidence in clinical research: a review of methods and applications. Res Synth Methods 2015;6(1):45-62. doi: 10.1002/jrsm.1122.
7. Schunemann H, Morgan R, Cuello C, Santesso N, Guyatt G, Verbeek J. Using GRADE to integrate randomised and non-randomised studies in systematic reviews. Abstracts of the Global Evidence Summit, Cape Town, South Africa. Cochrane Database of Systematic Reviews 2017, Issue 9 (Suppl 2). dx.doi.org/10.1002/14651858.CD201702.
Competing interests:
No competing interests
11 October 2017
Long Ge
PhD candidate
Jin-hui Tian associate professor, Ke-hu Yang professor
The First Clinical Medical College of Lanzhou University; Evidence-Based Medicine Center of Lanzhou University
No. 199, Donggang West Road, Chengguan District, Lanzhou City, China
Rapid Response:
AMSTAR 2: more considerations should be addressed
We have read and used AMSTAR 2 with great interest [1]. This instrument has made a major revision based on the original AMSTAR and will be used to assess systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. However, when we use the AMSTAR 2 instrument to assess the quality of systematic reviews including both randomised and non-randomised studies, debates on some items have been highlighted.
Item 2 (a protocol prior to conduct), it’s extremely important to develop a protocol of systematic review in advance. AMSTAR 2 answers this item as “Yes”, “Partial Yes”, and “No”. However, only a small minority of published non-Cochrane reviews reported a protocol [2]. When a protocol of systematic review doesn’t exist, “Not applicable” or “No protocol available” should be considered.
Item 4 (Literature search strategy), AMSTAR 2 has considered all key contents except for the reliability of the literature search. Methodological studies and guidelines have highlighted the importance of searchers, involving a local healthcare librarian or information specialist could effectively avoid errors during the conduct of the search [3-5]. Therefore, it’s important to determine the reliability of search by identifying whether systematic reviewers involve an information specialist or whether the search strategies were peer-reviewed.
Item 7 (provide a list of excluded studies and justify the exclusions), this item should contain two parts, one is a list of excluded studies, and another is the reasons for exclusion. Our experience tell us most published non-Cochrane reviews don’t provide the list of excluded studies, but give a reason of exclusion and (or) a c. When systematic reviews just report the reason of exclusion and (or) a PRISMA flow graph, should evidence users answer this item as “Partial Yes” or “No”?
Item 11 (appropriate methods for statistical combination), AMSTAR 2 indicates that authors should report pooled estimates separately for the different study types. In fact, some systematic reviews including both randomised and non-randomised studies combine the estimates from two study types. And there are some methodological studies to support the combination of evidence from randomised and non-randomised studies [6,7]. However, it’s more important to justify the combination methods for different study types.
Item 15 (investigation of publication bias), quantitative synthesis is performed, authors should investigate the likelihood and magnitude of publication bias. “Yes” are answered when performing graphical or statistical tests for publication bias. However, the Cochrane Handbook doesn’t recommend performing graphical or statistical tests for publication bias when the number of included studies is less than 10. In this situation, how do we assess this item?
The original AMSTAR has been a popular instrument for critically assessing systematic reviews of randomised studies. We believe that the current updated version will be more popular and more operable for researchers to assess the quality of systematic reviews and for evidence users to identify high quality systematic reviews.
Competing interests: No competing interests
References
1. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017,358:j4008. doi: 10.1136/bmj.j4008.
2. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med 2007, 4, e78.
3. Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.
4. Li L, Tian J, Tian H, et al. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol. 2014, 67:1001-7. doi: 10.1016/j.jclinepi.2014.04.003.
5. Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol 2009;62: 944e52. doi: 10.1016/j.jclinepi.2008.10.012.
6. Verde PE, Ohmann C. Combining randomized and non-randomized evidence in clinical research: a review of methods and applications. Res Synth Methods 2015;6(1):45-62. doi: 10.1002/jrsm.1122.
7. Schunemann H, Morgan R, Cuello C, Santesso N, Guyatt G, Verbeek J. Using GRADE to integrate randomised and non-randomised studies in systematic reviews. Abstracts of the Global Evidence Summit, Cape Town, South Africa. Cochrane Database of Systematic Reviews 2017, Issue 9 (Suppl 2). dx.doi.org/10.1002/14651858.CD201702.
Competing interests: No competing interests