Original ContributionsEvaluating the quality of systematic reviews in the emergency medicine literature*,**,*,**,♢
Introduction
Over the past 10 years, systematic reviews have gained popularity in the medical field as a method of synthesizing vast amounts of research evidence. The term “systematic review” is an umbrella term used to denote reviews that perform comprehensive searches designed to reduce selection and publication bias and also to evaluate included study quality. Not all systematic reviews pool data in a quantitative fashion (a so-called meta-analysis), but many do. Those that do not provide a pooled estimate are referred to as nonquantitative systematic reviews.
Therapy systematic reviews attempt to combine the results of individual clinical trials for the purposes of resolving conflict in the literature, enlarging sample sizes to increase statistical power, and investigating variations in different subgroups.1 Although a large number of randomized clinical trials have been published, they are often small and insufficiently powered to detect important treatment benefits.2 Therefore, systematic reviews can identify pooled results that clarify the evidence from the published literature.1 They are critical tools with outcomes that can serve to lessen the gap between evidence and clinical practice in medical care, especially in fields like emergency medicine, in which large multicenter trials are infrequently performed.3
However, there is heated debate about the value of systematic reviews. Many believe that they represent the highest level of evidence on which to base treatment recommendations at or above the level of large randomized controlled trials.4 Others are more critical and suggest that large clinical trials are the criterion standard to evaluate and establish effectiveness.5 Still others point to the variable quality of systematic reviews, an observation made frequently in the randomized controlled trial literature. One approach to resolving this debate is to find, produce, and disseminate scientifically valid systematic reviews.
The scientific quality of systematic reviews has been the source of considerable research in the past. Although some controversy still remains, most would agree that the scientific quality of a review is closely linked to reducing the biases inherent in this form of research. Several important sources of bias have been identified and studied: selection, publication, and pooling biases. If these influences are not carefully considered and minimized, a systematic review can lead to invalid results and conclusions.5, 6 Publication bias can occur when nonsignificant studies remain unpublished or when the authors of the systematic review do not perform a thorough literature search. Selection bias arises when selection of studies for inclusion in the review is influenced by factors such as the results, the authors, or some other factor. Systematic reviews have also been criticized for pooling data from a heterogeneous sample of studies (with varying case mixes, disease severities, or comorbidities). Avoiding errors in systematic reviews requires specific methodologic approaches, and assessing the validity of the results requires explicit reporting of the methods used to conduct the review.7
Although emergency medicine journals have not traditionally focused on this form of research, the frequency of publication is increasing. The objective of this study was to examine the scientific quality of systematic reviews published in the leading emergency medicine journals. To date, a quantitative evaluation of systematic reviews published in emergency medicine journals has not been reported, although comparisons between Cochrane reviews and articles published in paper-based journals have been completed.8 Research of this type is necessary to assist clinicians in the continual provision of evidence-based medicine.
Section snippets
Materials and methods
An expert search strategy was adapted from previous research9 and applied to both MEDLINE and EMBASE databases. The emergency medicine journals selected for searching were the Journal of Emergency Medicine (JEM) , American Journal of Emergency Medicine (AJEM) , Annals of Emergency Medicine (Annals) , Journal of Accident and Emergency Medicine (JAEM) , and Academic Emergency Medicine (AEM). An electronic search was performed and supplemented with hand searching in 4 of these journals (Annals ,
Results
The MEDLINE search (Table 1) identified 29 potential review articles published in the 5 journals between January 1988 and December 1998, of which 23 (79%) were eligible for the study. A similar EMBASE search identified 36 potential review articles within the same time period, of which 18 (50%) were eligible for the study. Of the eligible review articles identified in MEDLINE, 74% of them were also found in EMBASE, and of the eligible review articles identified in EMBASE, 95% of them were also
Discussion
This is the first study to evaluate the quality of systematic reviews in the emergency medicine literature. The results of this study suggest that searching for systematic reviews in the emergency medicine literature can be a difficult task. For example, multiple search strategies are required, and computerized searches, which are thought to be sensitive, miss many systematic reviews. Practically, this creates problems for clinicians attempting to identify high-quality systematic reviews for
Acknowledgements
Author contributions: KDK and BHR conceived the study and designed the protocol. MD and LS contributed to the search criteria and the quality scoring of any searches described in separate meta-analysis. KDK, AT, and BHR completed the independent OQAQ scoring; KDK and BHR independently completed data extraction. KDK supervised the entry and analysis of the data. KDK and BHR drafted the manuscript, and all authors contributed substantially to its revision. KDK and BHR take responsibility for the
References (48)
- et al.
Strengths and limitations of meta-analysis: larger studies may be more reliable
Control Clin Trials
(1997) - et al.
Agreement among reviewers of review articles
J Clin Epidemiol
(1991) - et al.
Validation of an index of the quality of review articles
J Clin Epidemiol
(1991) - et al.
A meta-analysis of blunt cardiac trauma: ending myocardial confusion
Am J Emerg Med
(1996) Adjunctive dexamethasone therapy for pediatric bacterial meningitis
J Emerg Med
(1996)Antibiotics to prevent infection in patients with dog bite wounds: a meta-analysis of randomized trials
Ann Emerg Med
(1994)- et al.
Antibiotics to prevent infection of simple wounds: a meta-analysis of randomized studies
Am J Emerg Med
(1995) - et al.
Cardiac arrest and resuscitation: a tale of 29 cities
Ann Emerg Med
(1990) - et al.
Cocaine-induced myocardial infarction: an analysis and review of the literature
J Emerg Med
(1992) Defibrillation by basic emergency medical technicians: effect on survival
Ann Emerg Med
(1995)
Effect of age on myocardial infarction and thrombolysis
Am J Emerg Med
Effect of out-of-hospital defibrillation by basic life support providers on cardiac arrest mortality: a metaanalysis
Ann Emerg Med
Effectiveness of emergency medical services for victims of out-of-hospital cardiac arrest: a metaanalysis
Ann Emerg Med
Steroid use in the emergency department treatment of asthma exacerbations: a meta-analysis
Am J Emerg Med
Epinephrine in cardiopulmonary resuscitation
J Emerg Med
Incidence of cardiac arrest: a neglected factor in evaluating survival rates
Ann Emerg Med
Life support courses: are they effective?
Ann Emerg Med
Physicochemical characteristics of drugs and response to repeat-dose activated charcoal
Am J Emerg Med
Practice guideline for the management of infants and children 0 to 36 months of age with fever without source
Ann Emerg Med
Steroids for the treatment of corrosive esophageal injury: a statistical analysis of past studies
Am J Emerg Med
A pooled analysis of the Ottawa Ankle rules used on adults in the ED
Am J Emerg Med.
Ultrasonography for the initial evaluation of blunt abdominal trauma: a review of prospective trials
Ann Emerg Med
Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?
Lancet
Meta-analyses to evaluate analgesic interventions: a systematic qualitative review of their methodology
J Clin Epidemiol
Cited by (70)
Quality of systematic reviews in African emergency medicine: a cross-sectional methodological study
2023, African Journal of Emergency MedicineDeclining Quality of Systematic Reviews in Orthopaedic Sports Medicine: An Updated Systematic Review
2022, Arthroscopy, Sports Medicine, and RehabilitationIs there any benefit in associating neuraxial anesthesia to general anesthesia for coronary artery bypass graft surgery?
2016, Brazilian Journal of AnesthesiologySome improvements are apparent in identifying adverse effects in systematic reviews from 1994 to 2011
2013, Journal of Clinical EpidemiologyCitation Excerpt :Detailed reporting of the full literature search process for systematic reviews is recommended and can impact on the reader's confidence in the results and conclusions [2–5]. Previous research has indicated deficiencies in the search quality and reporting of search strategies in systematic reviews [6–32], including systematic reviews of adverse effects [33–36]. Although research has indicated improvements over time in the overall quality of effectiveness reviews [37,38] and in the search methodology in dental systematic reviews [39], research on systematic reviews of adverse effects has not examined any time trends with respect to search quality and reporting [33–36].
Neuraxial Anesthesia Compared to General Anesthesia for Procedures on the Lower Half of the Body: Systematic Review of Systematic Reviews
2012, Revista Brasileira de Anestesiologia
- *
Dr. Kelly is currently affiliated with the Department of Rural Health, University of Melbourne, Shepparton, Victoria, Australia.
- **
Author contributions are provided at the end of this article.
- *
This study was supported by the Division of Emergency Medicine, University of Alberta (Dr. Rowe and Dr. Travers) in Edmonton, Alberta, Canada. Dr. Kelly has received funding from the Research Excellence Envelope in the Faculty of Medicine and Dentistry at the University of Alberta.
- **
The authors declare no known conflicts of interest. Dr. Kelly, Dr. Travers, and Dr. Rowe are active members of the Airway Review Group of the Cochrane Collaboration. Dr. Rowe is an author of one of the systematic reviews included in this study. Dr. Rowe has received research funding in the past from the following industry sponsors: Astra Pharma, Glaxo Wellcome, Pharmacia and Upjohn, Trudell, and Janssen-Ortho, but is not a paid consultant to any of these companies.
- ♢
Reprints not available from the authors.