Elsevier

The Lancet

Volume 354, Issue 9193, 27 November 1999, Pages 1896-1900
The Lancet

Review
Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement

https://doi.org/10.1016/S0140-6736(99)04149-5Get rights and content

Summary

Background

The Quality of Reporting of Meta-analyses (QUOROM) conference was convened to address standards for improving the quality of reporting of meta-analyses of clinical randomised controlled trials (RCTs).

Methods

The QUOROM group consisted of 30 clinical epidemiologists, clinicians, statisticians, editors, and researchers. In conference, the group was asked to identify items they thought should be included in a checklist of standards. Whenever possible, checklist items were guided by research evidence suggesting that failure to adhere to the item proposed could lead to biased results. A modified Delphi technique was used in assessing candidate items.

Findings

The conference resulted in the QUOROM statement, a checklist, and a flow diagram. The checklist describes our preferred way to present the abstract, introduction, methods, results, and discussion sections of a report of a meta-analysis. It is organised into 21 headings and subheadings regarding searches, selection, validity assessment, data abstraction, study characteristics, and quantitative data synthesis, and in the results with “trial flow”, study characteristics, and quantitative data synthesis; research documentation was identified for eight of the 18 items. The flow diagram provides information about both the numbers of RCTs identified, included, and excluded and the reasons for exclusion of trials.

Interpretation

We hope this report will generate further thought about ways to improve the quality of reports of meta-analyses of RCTs and that interested readers, reviewers, researchers, and editors will use the QUOROM statement and generate ideas for its improvement.

Introduction

Health-care providers and other decision-makers now have, among their information resources, a form of clinical report called the meta-analysis,1, 2, 3, 4 a review in which bias has been reduced by the systematic identification, appraisal, synthesis, and, if relevant, statistical aggregation of all relevant studies on a specific topic according to a predetermined and explicit method.3 The number of published meta-analyses has increased substantially in the past decade.5 These integrative articles can be helpful for clinical decisions, and they may also serve as the policy foundation for evidence-based practice guidelines, economic evaluations, and future research agendas. The value of meta-analysis is evident in the work of the international Cochrane Collaboration,6, 7 the primary purpose of which is to generate and disseminate high-quality systematic reviews of health-care interventions.

Like any research enterprise, particularly one that is observational, the meta-analysis of evidence can be flawed. Accordingly, the process by which meta-analyses are carried out has undergone scrutiny. A 1987 survey of 86 English-language meta-analyses8 assessed each publication on 23 items from six content areas judged important in the conduct and reporting of a meta-analysis of randomised trials: study design, combinability, control of bias, statistical analysis, sensitivity analysis, and problems of applicability. The survey results showed that only 24 (28%) of the 86 meta-analyses reported that all six content areas had been addressed. The updated survey, which included more recently published meta-analyses, showed little improvement in the rigour with which they were reported.9

Several publications have described the science of reviewing research,1 differences among narrative reviews, systematic reviews, and meta-analyses,2 and how to carry out,3, 4, 10 critically appraise,11, 12, 13, 14, 15 and apply16 meta-analyses in practice. The increase in the number of meta-analyses published has highlighted such issues as discordant meta-analyses on the same topic17 and discordant meta-analyses and randomised-trial results on the same question.18

An important consideration in interpretation and use of meta-analyses is to ascertain that the investigators who did the meta-analysis not only report explicitly the methods they used to analyse the articles they reviewed, but also report the methods used in the research articles they analysed. The meta-analytical review methods used may not be provided when a paper is initially submitted: even when they are, other factors such as page limitations, peer review, and editorial decisions may change the content and format of the report before publication.

Several investigators have suggested guidelines for reporting of meta-analyses.3, 19 However, a consensus across disciplines has not developed. After the initiative to improve the quality of reporting of randomised controlled trials (RCTs),20, 21, 22 we organised the Quality of Reporting of Meta-analyses (QUOROM) conference to address these issues as they relate to meta-analyses of RCTs. This report summarises the proceedings of that conference. The issues discussed might also be useful for reporting of systematic reviews (ie, meta-analysis, as defined above, without statistical aggregation), particularly of RCTs.

Section snippets

Methods

The QUOROM steering committee began with a comprehensive review of publications on the conduct and reporting of meta-analyses. The databases searched included MEDLINE and the Cochrane Library,23 which consists of the Cochrane Database of Systematic Reviews, the Cochrane Controlled Trials Register, the York Database of Abstracts of Reviews of Effectiveness, and the Cochrane Review Methodology Database. We examined reference lists of the retrieved articles and individual personal files. Articles

Results

The conference resulted in the QUOROM statement: a checklist (table) and a flow diagram (figure). The checklist of standards for reporting of meta-analyses describes our preferred way to present the abstract, introduction, methods, results, and discussion sections of a report of a meta-analysis. The checklist is organised into 21 headings and subheadings to encourage authors to provide readers with information on searches, selection, validity assessment, data abstraction, study characteristics,

Discussion

In developing the checklist, we identified supporting scientific evidence for only eight of 18 items to guide the reporting of meta-analyses of RCTs.26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39 Some of this evidence is indirect. For example, we ask authors to use a structured abstract format. The supporting evidence for this item was collected by examining abstracts of original reports of individual studies27 and may not pertain specifically to the reporting of meta-analyses. However,

References (48)

  • D Moher et al.

    Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews

    Lancet

    (1996)
  • A Vickers et al.

    Do certain countries produce only positive results? A systematic review of controlled trials

    Control Clin Trials

    (1998)
  • A Davies et al.

    Risperidone versus haloperidol: I, meta-analysis of efficacy and safety

    Clin Ther

    (1998)
  • S Leucht et al.

    Efficacy and extrapyramidal side-effects of the new antipsychotics olanzapine, quetiapine, risperidone, and sertindole compared to conventional antipsychotics and placebo: a meta-analysis of randomized controlled trials

    Schizophrenia Res

    (1999)
  • CD Mulrow

    The medical review article: state of the science

    Ann Intern Med

    (1987)
  • DJ Cook et al.

    Systematic reviews: synthesis of best evidence for clinical decisions

    Ann Intern Med

    (1997)
  • J Deeks et al.

    Undertaking systematic reviews of research on effectiveness CRD guidelines for those carrying out or commissioning reviews. CRD report no 4

    (1996)
  • I Chalmers et al.

    Reporting, updating, and correcting systematic reviews of the effects of health care

  • L Bero et al.

    The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care

    JAMA

    (1995)
  • P Huston

    The Cochrane Collaboration helping unravel tangled web woven by international research

    Can Med Assoc J

    (1996)
  • HS Sacks et al.

    Meta-analyses of randomized controlled trials

    N Engl J Med

    (1987)
  • HS Sacks et al.

    Meta-analysis: an update

    Mt Sinai J Med

    (1996)
  • AD Oxman et al.

    Users' guides to the medical literature: VI, how to use an overview

    JAMA

    (1994)
  • Cited by (4039)

    View all citing articles on Scopus

    Other members listed at end of paper

    View full text