Systematic reviews and meta-analyses on treatment of asthma: critical evaluationBMJ 2000; 320 doi: https://doi.org/10.1136/bmj.320.7234.537 (Published 26 February 2000) Cite this as: BMJ 2000;320:537
- Alejandro R Jadad, professor ()a,
- Michael Moher, Royal College of General Practitioners research training fellowb,
- George P Browman, professora,
- Lynda Booker, research assistanta,
- Christopher Sigouin, doctoral studenta,
- Mario Fuentes, research assistantc,
- Robert Stevens, research assistantc
- a Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Canada L8N 3Z5
- b Institute of Health Sciences, University of Oxford, Headington, Oxford OX3 7LF
- c Foresight Consultants, Dundas, Ontario, Canada L9H 2R5
- Correspondence to: A R Jadad
- Accepted 13 December 1999
Objective: To evaluate the clinical, methodological, and reporting aspects of systematic reviews and meta-analyses on the treatment of asthma and to compare those published by the Cochrane Collaboration with those published in paper based journals.
Design: Analysis of studies identified from Medline, CINAHL, HealthSTAR, EMBASE, Cochrane Library, personal collections, and reference lists.
Studies: Articles describing a systematic review or a meta-analysis of the treatment of asthma that were published as a full report, in any language or format, in a peer reviewed journal or the Cochrane Library.
Main outcome measures: General characteristics of studies reviewed and methodological characteristics (sources of articles; language restrictions; format, design, and publication status of studies included; type of data synthesis; and methodological quality).
Results: 50 systematic reviews and meta-analyses were included. More than half were published in the past two years. Twelve reviews were published in the Cochrane Library and 38 were published in 22 peer reviewed journals. Forced expiratory volume in one second was the most frequently used outcome, but few reviews evaluated the effect of treatment on costs or patient preferences. Forty reviews were judged to have serious or extensive flaws. All six reviews associated with industry were in this group. Seven of the 10 most rigorous reviews were published in the Cochrane Library.
Conclusions: Most reviews published in peer reviewed journals or funded by industry have serious methodological flaws that limit their value to guide decisions. Cochrane reviews are more rigorous and better reported than those published in peer reviewed journals.
One of the most important challenges for anyone treating patients with asthma is the need to cope with vast amounts of information about an ever increasing number of new devices and drugs. The abundance of information often makes it difficult to establish the most appropriate drug or regimen to use in a given clinical situation.1
During the past two decades, interest has risen in the use of systematic reviews as tools to help decision makers cope with information overload. These reviews, in their ideal form, include an explicit description of how they were conducted and incorporate strategies to minimise bias and maximise precision. 2 3 These reviews can also include meta-analysis (the statistical combination of the results of several independent studies to produce a single estimate of the effect of a particular intervention or healthcare situation).4 However, most systematic reviews and meta-analyses published in peer reviewed journals have methodological deficiencies that may limit their validity.5–12
Recently, important initiatives have emerged to improve the methodological quality of systematic reviews and meta-analyses. Perhaps the most prominent is the Cochrane Collaboration, a large international group 11 13 that seems to be producing reviews that are more rigorous and better reported than those published in peer reviewed journals. 9 14 We evaluated the quality of published systematic reviews and meta-analyses on the treatment of asthma and compared the characteristics of Cochrane reviews with those in peer reviewed journals.
To be included, a report had to be described as a systematic review or a meta-analysis of the treatment of asthma and to be published in full, in any language or format, in a journal or the Cochrane Library.
Literature search and study selection
We identified eligible studies by a search up to July 1998 in Medline (from 1966), CINAHL (from 1982), HealthSTAR (from 1975), and EMBASE (from 1984) using a refined search strategy (see BMJ's website for details).9 This was complemented by a search of the Cochrane Library (issue 3, 1998) with the term “asthma” and searches of personal collections and the reference lists of eligible studies.
Independently, two of us (LB and ARJ) reviewed the citations identified by the search strategy to judge their eligibility. The final decision on inclusion in the review was made by consensus.
Three trained reviewers (MF, LB, RS) extracted data independently using unmasked reports.15 The final set was agreed by consensus, and differences were resolved by a fourth reviewer (ARJ).
Data were extracted on the general characteristics of the studies (authors, sources of funding, and source, year, place, and language of publication); clinical issues (population, intervention, condition, and outcomes reviewed), methodological characteristics (language restrictions; number, format, design, and publication status of the studies included; data synthesis; heterogeneity testing; and methodological quality); and conclusions. Methodological quality was assessed with Oxman and Guyatt's index, a validated tool that scores reviews on a seven point scale. 8 16 Reports were regarded as having serious or extensive flaws if they received a score of 1 to 3, and as having minimal or minor flaws if they received scores from 4 to 7.
Reviews were divided according to publication in the Cochrane Library or journals and to the presence or absence of association with industry. The Wilcoxon rank-sum test (two sided) was used to compare the number of authors, included studies, and sources and the overall quality scores. Fisher's exact test (two sided) was used to compare the responses to the first nine items in Oxman and Guyatt's index and also the proportion of reviews that described the primary outcomes, selection criteria, and heterogeneity testing and the number of reviews with language restrictions between the Cochrane Library and journals. Values of P<0.05 were regarded as significant.
We identified 50 eligible reviews (see BMJ's website for full list). The first review identified was published in 1988. Twenty nine (58%) of the reviews were published in 1997 and 1998. Twelve reviews were published in the Cochrane Library, and the remaining 38 were published in 22 peer reviewed journals. Only one review appeared in both groups. None of the reviews published in journals represented updated versions of earlier work. The journal with the highest number of reviews was Annals of Pharmacotherapy (seven) followed by the Journal of Allergy and Clinical Immunology and Respiratory Medicine (four each). Table 1 gives other characteristics of the reviews.
The number of studies included in the reviews varied from 2 to 97. Most of the reviews (37) included randomised controlled trials, but three included non-randomised trials with contemporaneous controls and one included case series. In 14 reviews (including some of those mentioned in the previous sentence) the exact type of studies was either unclear or not reported. Seven of the 50 reviews included unpublished articles and nine included articles available only in abstract form. None of the reviews published in peer reviewed journals had been updated.
Forty reviews obtained quality scores of 3 or less, which is compatible with serious or extensive flaws. All six reviews associated with industry were in this group (table 2). None of these reviews described efforts to reduce selection bias; one described the search strategy, one described the methods used to assess the quality of the studies, and three reported the methods used to combine data across studies (table 2). All but one17 of these studies had results and conclusions that favoured the interventions related to the companies sponsoring the reviews. The number of reviews associated with industry was too small to warrant meaningful statistical comparisons with other reviews.
Analysis of individual components of Oxman and Guyatt's index showed that 66% of the reviews stated the search methods used to find the evidence, 52% included a reasonably comprehensive search for evidence, 56% reported the inclusion criteria, 30% included measures to avoid selection bias, 28% reported the criteria used to assess the validity of the included studies, 24% used appropriate criteria for validity assessment, 52% reported the methods used to synthesise findings, and 50% had conclusions supported by the data or the analysis reported in the review (table 2). In addition, 20 reviews (40%) included heterogeneity testing, 16 (32%) stated the primary outcome of interest to the reviewers, and eight (16%) included assessment of publication bias. We found no significant difference in the proportion of reviews with favourable results between those with minimal or minor flaws and those with serious or extensive flaws (1/10 v 11/40, P=0.416).
Cochrane reviews versus reviews in peer reviewed journals
Of the 10 reviews with scores of 4 or more, seven were published in the Cochrane Library. Cochrane reviews had higher overall quality scores than those published in peer reviewed journals (median 6 v 2, P<0.005). Compared with reviews published in journals, more Cochrane reviews had adequate reporting of the search strategy, had comprehensive search strategies, described the inclusion criteria, made efforts to avoid bias during the selection of the studies, reported the criteria for assessing the quality of the studies, reported the methods used to combine study findings, and combined the findings of the relevant studies appropriately (table 2). Fewer Cochrane reviews had language restrictions (0/12 v 21/38, P<0.005) and more included heterogeneity testing (9 v 11, P=0.007).
There were no significant differences between the groups in the number of authors, the number of studies reviewed, assessment of publication bias, and identification of primary outcomes.
Findings of reviews
Twelve reviews identified an intervention as better than the control treatment. None of the reviews reported patient preferences. The box gives the conclusions of the reviews with minimal flaws.
Findings of studies with minimal flaws
Peer reviewed journals
Use of oral and intravenous steroids early in the treatment of asthma exacerbations reduces hospital admissions in adults and children; steroids are effective in preventing relapse in outpatient management of exacerbations; and oral and intravenous steroids seem to have equivalent effects on pulmonary function during acute exacerbations18
Addition of ipratropium bromide to β agonist significantly improves the predicted forced expiratory volume in one second but gives no clinical improvement19
Bronchodilator delivery through a meter dosed inhaler or wet nebuliser is equivalent in acute treatment of asthma20
Limited asthma education is unlikely to improve health outcomes in adults with asthma21
Addition of a single inhalation of anticholinergics to a β agonist regimen may improve lung function but has no notable effect on hospital admissions. Multiple doses may avoid hospital admission in 1 of 11 treated patients22
A short course of corticosteroids (oral or intramuscular) after an acute exacerbation of asthma (of variable and poorly reported severity at baseline) reduces the number of relapses and decreases use of β agonist without apparent increase in adverse effects23
There is no indication for routine use of methotrexate in patients receiving long term treatment with oral steroids24
Prophylaxis with single doses of nedocromil sodium1-8 mg inhibits the severity and duration of exercise induced bronchoconstriction. Benefits seem greatest in patients with more severe symptoms25
Allergen specific immunotherapy reduces the symptoms and drug requirements but has no consistent effect on lung function. The effects are greater in patients with allergen specific bronchial hyperreactivity than in those with non-specific hyperreactivity26
Chemical and physical measures to reduce exposure to mites or their products at home seem to be ineffective27
The number of systematic reviews and meta-analyses evaluating the treatment of asthma is increasing. Most of the reviews have clinical relevance but have methodological limitations that could have been avoided easily. Future reviews should include comprehensive and well reported search strategies, attempts to reduce bias during the selection of the studies, assessment of the validity of the studies, and clear description of the methods used to synthesise the data available. Few reviews address safety issues or the effect of treatment on quality of life, costs, or patient preferences.
This study also confirms that Cochrane reviews are more rigorous than reviews published in peer reviewed journals. 9 14 Several factors may explain the differences. Cochrane reviews follow standardised instructions, are peer reviewed at several levels, and receive input from different groups at different stages.13 One limitation of our analyses is that the tool used for quality assessments was developed by Andrew Oxman, who led the development of the methodology of Cochrane reviews.
Perhaps peer reviewed journals could improve the quality of the systematic reviews they publish by providing authors and peer reviewers with clear reporting criteria, including those followed by Cochrane reviewers, 28 29 by influencing the review process at a much earlier stage, and by promoting more frequent updates or correction of published material on the internet. Authors should be particularly careful when doing reviews in association with industry, as such reviews have been shown to have an increased risk of producing results in favour of the interventions promoted by the sponsors.12 This was the case for all but one of the reviews associated with industry in our study. The only review that did not favour an intervention was designed to evaluate the effects of vitamin C on asthma, not a new proprietary compound.17
Bridging the gap between methodological research and methodological practice will require unprecedented efforts by researchers, consumers, funders, journal editors, and peer reviewers. Efforts made by members of the Cochrane Collaboration are already contributing to filling the gap and should be emulated by peer reviewed journals.
What is already known on this topic
Systematic reviews and meta-analyses could help decision makers cope with information overload
The number of systematic reviews and meta-analyses evaluating asthma treatments is increasing
What this study adds
Most reviews of asthma treatment published in peer reviewed journals or funded by industry have serious methodological flaws that limit their usefulness
Cochrane reviews are more rigorous and better reported than those published in peer reviewed journals
The clinically relevant messages from the 10 most rigorous reviews are summarised
We thank Ann Murray for editorial assistance. The review was coordinated by Foresight Consultants, a division of Foresight Links Corporation.
Contributors: ARJ (guarantor) initiated and designed the study, participated in data collection and analysis, wrote the first draft of the paper, collated comments from the other authors, and incorporated the comments of the peer reviewers into the final version. MM and GB provided input to the design of the study, contributed to the design of the data extraction forms, and commented on all the drafts of the paper. LB, MF, and RS provided input to the data extraction forms and extracted the data from the reviews. CS was responsible for data checking and the statistical analysis.
Funding Foresight Consultants was commissioned by AstraZeneca Research and Development Lund, Sweden, and AstraZeneca US to perform an independent review of this topic through an unrestricted contract. Representatives from AstraZeneca did not participate in the study selection or appraisal or in the analysis of the data. ARJ was supported in part by a National Health Research Scholar Award from Health Canada, Ottawa, Ontario.
Competing interests ARJ is codirector of the Canadian Cochrane Network and Centre.
website extra: Details of the search strategy and all included studies are available on the BMJ's website www.bmj.com