BMJ  2005;330:1053 (7 May), doi:10.1136/bmj.38414.515938.8F (published 7 April 2005)

Paper

Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study

¹ Centro Cochrane Italiano, Istituto Mario Negri, Via Eritrea 62, 20157 Milan, Italy, ² Università degli Studi di Modena e Reggio Emilia, Modena, Italy

Objectives To describe how the methodological quality of primary studies is assessed in systematic reviews and whether the quality assessment is taken into account in the interpretation of results.

Data sources Cochrane systematic reviews and systematic reviews in paper based journals.

Study selection 965 systematic reviews (809 Cochrane reviews and 156 paper based reviews) published between 1995 and 2002.

Data synthesis The methodological quality of primary studies was assessed in 854 of the 965 systematic reviews (88.5%). This occurred more often in Cochrane reviews than in paper based reviews (93.9% v 60.3%, P < 0.0001). Overall, only 496 (51.4%) used the quality assessment in the analysis and interpretation of the results or in their discussion, with no significant differences between Cochrane reviews and paper based reviews (52% v 49%, P = 0.58). The tools and methods used for quality assessment varied widely.

Conclusions Cochrane reviews fared better than systematic reviews published in paper based journals in terms of assessment of methodological quality of primary studies, although they both largely failed to take it into account in the interpretation of results. Methods for assessment of methodological quality by systematic reviews are still in their infancy and there is substantial room for improvement.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Articles

Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement

David Moher, Alessandro Liberati, Jennifer Tetzlaff, Douglas G Altman for the PRISMA Group
BMJ 2009 339: b2535. [Full Text]

The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration

Alessandro Liberati, Douglas G Altman, Jennifer Tetzlaff, Cynthia Mulrow, Peter C Gøtzsche, John P A Ioannidis, Mike Clarke, P J Devereaux, Jos Kleijnen, and David Moher
BMJ 2009 339: b2700. [Abstract] [Full Text]

Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study

Lesley Wood, Matthias Egger, Lise Lotte Gluud, Kenneth F Schulz, Peter Jüni, Douglas G Altman, Christian Gluud, Richard M Martin, Anthony J G Wood, and Jonathan A C Sterne
BMJ 2008 336: 601-605. [Abstract] [Full Text] [PDF]

Effects of acupuncture on rates of pregnancy and live birth among women undergoing in vitro fertilisation: systematic review and meta-analysis

Eric Manheimer, Grant Zhang, Laurence Udoff, Aviad Haramati, Patricia Langenberg, Brian M Berman, and Lex M Bouter
BMJ 2008 336: 545-549. [Abstract] [Full Text] [PDF]

Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review

Anders W Jørgensen, Jørgen Hilden, and Peter C Gøtzsche
BMJ 2006 333: 782. [Abstract] [Full Text] [PDF]

Systematic reviews in health care: Assessing the quality of controlled clinical trials

Peter Jüni, Douglas G Altman, and Matthias Egger
BMJ 2001 323: 42-46. [Extract] [Full Text] [PDF]

Systematic reviews and meta-analyses on treatment of asthma: critical evaluation

Alejandro R Jadad, Michael Moher, George P Browman, Lynda Booker, Christopher Sigouin, Mario Fuentes, and Robert Stevens
BMJ 2000 320: 537-540. [Abstract] [Full Text] [PDF]

The miracle of DICE therapy for acute stroke: fact or fictional product of subgroup analysis?

Carl E Counsell, Mike J Clarke, Jim Slattery, and Peter A G Sandercock
BMJ 1994 309: 1677-1681. [Abstract] [Full Text]

This article has been cited by other articles:

• Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., and the PRISMA Group, (2009). Reprint--Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. ptjournal 89: 873-880 [Abstract] [Full text]  
• Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., the PRISMA Group, (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. ANN INTERN MED 151: 264-269 [Full text]  
• Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P.A., Clarke, M., Devereaux, P. J., Kleijnen, J., Moher, D. (2009). The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration. ANN INTERN MED 151: W-65-W-94 [Abstract] [Full text]  
• Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G, for the PRISMA Group, (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339: b2535-b2535 [Full text]  
• Liberati, A., Altman, D. G, Tetzlaff, J., Mulrow, C., Gotzsche, P. C, Ioannidis, J. P A, Clarke, M., Devereaux, P J, Kleijnen, J., Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 339: b2700-b2700 [Abstract] [Full text]  
• Diaz-Insua, M. (2009). Review: Methodological quality in medical evidence, quo vadis?. Therapeutic Advances in Urology 1: 51-59 [Abstract]  
• Brok, J., Thorlund, K., Wetterslev, J., Gluud, C. (2009). Apparently conclusive meta-analyses may be inconclusive--Trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analyses. Int J Epidemiol 38: 287-298 [Abstract] [Full text]  
• Doi, S. A. R. (2008). Low dose aspirin and IVF: 'Is it time for a meta-analysis' continued?. Hum Reprod Update 14: 537-538 [Full text]  
• Wykes, T., Steel, C., Everitt, B., Tarrier, N. (2008). Cognitive Behavior Therapy for Schizophrenia: Effect Sizes, Clinical Models, and Methodological Rigor. Schizophr Bull 34: 523-537 [Abstract] [Full text]  
• Brok, J, Greisen, G, Madsen, L P, Tilma, K, Faerk, J, Borch, K, Garne, E, Christesen, H T, Stanchev, H, Jacobsen, T, Nielsen, J P, Henriksen, T B, Gluud, C (2008). Agreement between Cochrane Neonatal reviews and clinical practice guidelines for newborns in Denmark: a cross-sectional study. Arch. Dis. Child. Fetal Neonatal Ed. 93: F225-F229 [Abstract] [Full text]  
• Wood, L., Egger, M., Gluud, L. L., Schulz, K. F, Juni, P., Altman, D. G, Gluud, C., Martin, R. M, Wood, A. J G, Sterne, J. A C (2008). Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 336: 601-605 [Abstract] [Full text]  
• Manheimer, E., Zhang, G., Udoff, L., Haramati, A., Langenberg, P., Berman, B. M, Bouter, L. M (2008). Effects of acupuncture on rates of pregnancy and live birth among women undergoing in vitro fertilisation: systematic review and meta-analysis. BMJ 336: 545-549 [Abstract] [Full text]  
• Fuccio, L., Minardi, M. E., Zagari, R. M., Grilli, D., Magrini, N., Bazzoli, F. (2007). Meta-analysis: Duration of First-Line Proton-Pump Inhibitor-Based Triple Therapy for Helicobacter pylori Eradication. ANN INTERN MED 147: 553-562 [Abstract] [Full text]  
• Pildal, J, Hrobjartsson, A, Jorgensen, K., Hilden, J, Altman, D., Gotzsche, P. (2007). Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 36: 847-857 [Abstract] [Full text]  
• Poolman, R. W., Sierevelt, I. N., Farrokhyar, F., Mazel, J. A., Blankevoort, L., Bhandari, M. (2007). Perceptions and Competence in Evidence-Based Medicine: Are Surgeons Getting Better? A Questionnaire Survey of Members of the Dutch Orthopaedic Association. JBJS 89: 206-215 [Abstract] [Full text]  
• Jorgensen, A. W, Hilden, J., Gotzsche, P. C (2006). Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ 333: 782- [Abstract] [Full text]  

Rapid Responses:

Read all Rapid Responses

EBM is flawed: not invalid, but flawed

Neville W Goodman
bmj.com, 17 May 2005 [Full text]

Online poll

Find out more on doc2doc >>