Original Contributions
Evaluating the quality of systematic reviews in the emergency medicine literature*,**,*,**,

Presented at the Society for Academic Emergency Medicine annual meeting, Boston, MA, May 1999.
https://doi.org/10.1067/mem.2001.115881Get rights and content

Abstract

Study Objective: The objective of this study was to examine the scientific quality of systematic reviews published in 5 leading emergency medicine journals. Methods: MEDLINE and EMBASE databases were electronically searched to identify published systematic reviews. Searches were only conducted in emergency medicine journals during the past 10 years; 4 of the journals were also hand searched. Potential reviews were assessed independently by 2 reviewers for inclusion. Data regarding methods were extracted from each review independently by 2 reviewers. All systematic reviews were retrieved and rated for quality by using the 10 questions from the overview quality assessment questionnaire. Results: Twenty-nine reviews were identified from more than 100 citations. The overall scientific quality of the systematic reviews was low (mean score, 2.7; 95% confidence interval 2.1 to 3.2; maximum possible score, 7.0). Selection and publication biases were rarely addressed in this collection of reviews. For example, the search strategies were only identified in 9 (31%) reviews, whereas independent study selection (6 [21%]) and quality assessment of included studies (9 [31%]) were infrequently performed. Overall, the majority of reviews had extensive flaws, and only 3 (10%) had minimal flaws. Conclusion: The results of the study indicate that many of the systematic reviews published in the emergency medicine literature contain major flaws; reviews with poor methodology may limit the validity of reported results. Further efforts should be made to improve the design, reporting, and publication of systematic reviews in emergency medicine. [Kelly KD, Travers A, Dorgan M, Slater L, Rowe BH. Evaluating the quality of systematic reviews in the emergency medicine literature. Ann Emerg Med. November 2001;38:518-526.]

Introduction

Over the past 10 years, systematic reviews have gained popularity in the medical field as a method of synthesizing vast amounts of research evidence. The term “systematic review” is an umbrella term used to denote reviews that perform comprehensive searches designed to reduce selection and publication bias and also to evaluate included study quality. Not all systematic reviews pool data in a quantitative fashion (a so-called meta-analysis), but many do. Those that do not provide a pooled estimate are referred to as nonquantitative systematic reviews.

Therapy systematic reviews attempt to combine the results of individual clinical trials for the purposes of resolving conflict in the literature, enlarging sample sizes to increase statistical power, and investigating variations in different subgroups.1 Although a large number of randomized clinical trials have been published, they are often small and insufficiently powered to detect important treatment benefits.2 Therefore, systematic reviews can identify pooled results that clarify the evidence from the published literature.1 They are critical tools with outcomes that can serve to lessen the gap between evidence and clinical practice in medical care, especially in fields like emergency medicine, in which large multicenter trials are infrequently performed.3

However, there is heated debate about the value of systematic reviews. Many believe that they represent the highest level of evidence on which to base treatment recommendations at or above the level of large randomized controlled trials.4 Others are more critical and suggest that large clinical trials are the criterion standard to evaluate and establish effectiveness.5 Still others point to the variable quality of systematic reviews, an observation made frequently in the randomized controlled trial literature. One approach to resolving this debate is to find, produce, and disseminate scientifically valid systematic reviews.

The scientific quality of systematic reviews has been the source of considerable research in the past. Although some controversy still remains, most would agree that the scientific quality of a review is closely linked to reducing the biases inherent in this form of research. Several important sources of bias have been identified and studied: selection, publication, and pooling biases. If these influences are not carefully considered and minimized, a systematic review can lead to invalid results and conclusions.5, 6 Publication bias can occur when nonsignificant studies remain unpublished or when the authors of the systematic review do not perform a thorough literature search. Selection bias arises when selection of studies for inclusion in the review is influenced by factors such as the results, the authors, or some other factor. Systematic reviews have also been criticized for pooling data from a heterogeneous sample of studies (with varying case mixes, disease severities, or comorbidities). Avoiding errors in systematic reviews requires specific methodologic approaches, and assessing the validity of the results requires explicit reporting of the methods used to conduct the review.7

Although emergency medicine journals have not traditionally focused on this form of research, the frequency of publication is increasing. The objective of this study was to examine the scientific quality of systematic reviews published in the leading emergency medicine journals. To date, a quantitative evaluation of systematic reviews published in emergency medicine journals has not been reported, although comparisons between Cochrane reviews and articles published in paper-based journals have been completed.8 Research of this type is necessary to assist clinicians in the continual provision of evidence-based medicine.

Section snippets

Materials and methods

An expert search strategy was adapted from previous research9 and applied to both MEDLINE and EMBASE databases. The emergency medicine journals selected for searching were the Journal of Emergency Medicine (JEM) , American Journal of Emergency Medicine (AJEM) , Annals of Emergency Medicine (Annals) , Journal of Accident and Emergency Medicine (JAEM) , and Academic Emergency Medicine (AEM). An electronic search was performed and supplemented with hand searching in 4 of these journals (Annals ,

Results

The MEDLINE search (Table 1) identified 29 potential review articles published in the 5 journals between January 1988 and December 1998, of which 23 (79%) were eligible for the study. A similar EMBASE search identified 36 potential review articles within the same time period, of which 18 (50%) were eligible for the study. Of the eligible review articles identified in MEDLINE, 74% of them were also found in EMBASE, and of the eligible review articles identified in EMBASE, 95% of them were also

Discussion

This is the first study to evaluate the quality of systematic reviews in the emergency medicine literature. The results of this study suggest that searching for systematic reviews in the emergency medicine literature can be a difficult task. For example, multiple search strategies are required, and computerized searches, which are thought to be sensitive, miss many systematic reviews. Practically, this creates problems for clinicians attempting to identify high-quality systematic reviews for

Acknowledgements

Author contributions: KDK and BHR conceived the study and designed the protocol. MD and LS contributed to the search criteria and the quality scoring of any searches described in separate meta-analysis. KDK, AT, and BHR completed the independent OQAQ scoring; KDK and BHR independently completed data extraction. KDK supervised the entry and analysis of the data. KDK and BHR drafted the manuscript, and all authors contributed substantially to its revision. KDK and BHR take responsibility for the

References (48)

  • B Morgan et al.

    Effect of age on myocardial infarction and thrombolysis

    Am J Emerg Med

    (1995)
  • TE Auble et al.

    Effect of out-of-hospital defibrillation by basic life support providers on cardiac arrest mortality: a metaanalysis

    Ann Emerg Med

    (1995)
  • G Nichol et al.

    Effectiveness of emergency medical services for victims of out-of-hospital cardiac arrest: a metaanalysis

    Ann Emerg Med

    (1996)
  • BH Rowe et al.

    Steroid use in the emergency department treatment of asthma exacerbations: a meta-analysis

    Am J Emerg Med

    (1992)
  • P Hebert et al.

    Epinephrine in cardiopulmonary resuscitation

    J Emerg Med

    (1991)
  • LB Becker et al.

    Incidence of cardiac arrest: a neglected factor in evaluating survival rates

    Ann Emerg Med

    (1993)
  • M Jabbour et al.

    Life support courses: are they effective?

    Ann Emerg Med

    (1996)
  • JW Campbell et al.

    Physicochemical characteristics of drugs and response to repeat-dose activated charcoal

    Am J Emerg Med

    (1992)
  • LJ Baraff et al.

    Practice guideline for the management of infants and children 0 to 36 months of age with fever without source

    Ann Emerg Med

    (1993)
  • JM Howell et al.

    Steroids for the treatment of corrosive esophageal injury: a statistical analysis of past studies

    Am J Emerg Med

    (1992)
  • RJ Markert et al.

    A pooled analysis of the Ottawa Ankle rules used on adults in the ED

    Am J Emerg Med.

    (1998)
  • WS Pearl et al.

    Ultrasonography for the initial evaluation of blunt abdominal trauma: a review of prospective trials

    Ann Emerg Med

    (1996)
  • D Moher et al.

    Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?

    Lancet

    (1998)
  • AR Jadad et al.

    Meta-analyses to evaluate analgesic interventions: a systematic qualitative review of their methodology

    J Clin Epidemiol

    (1996)
  • Cited by (70)

    • Some improvements are apparent in identifying adverse effects in systematic reviews from 1994 to 2011

      2013, Journal of Clinical Epidemiology
      Citation Excerpt :

      Detailed reporting of the full literature search process for systematic reviews is recommended and can impact on the reader's confidence in the results and conclusions [2–5]. Previous research has indicated deficiencies in the search quality and reporting of search strategies in systematic reviews [6–32], including systematic reviews of adverse effects [33–36]. Although research has indicated improvements over time in the overall quality of effectiveness reviews [37,38] and in the search methodology in dental systematic reviews [39], research on systematic reviews of adverse effects has not examined any time trends with respect to search quality and reporting [33–36].

    View all citing articles on Scopus
    *

    Dr. Kelly is currently affiliated with the Department of Rural Health, University of Melbourne, Shepparton, Victoria, Australia.

    **

    Author contributions are provided at the end of this article.

    *

    This study was supported by the Division of Emergency Medicine, University of Alberta (Dr. Rowe and Dr. Travers) in Edmonton, Alberta, Canada. Dr. Kelly has received funding from the Research Excellence Envelope in the Faculty of Medicine and Dentistry at the University of Alberta.

    **

    The authors declare no known conflicts of interest. Dr. Kelly, Dr. Travers, and Dr. Rowe are active members of the Airway Review Group of the Cochrane Collaboration. Dr. Rowe is an author of one of the systematic reviews included in this study. Dr. Rowe has received research funding in the past from the following industry sponsors: Astra Pharma, Glaxo Wellcome, Pharmacia and Upjohn, Trudell, and Janssen-Ortho, but is not a paid consultant to any of these companies.

    Reprints not available from the authors.

    View full text