Original Article
The reporting quality of meta-analyses improves: a random sampling study

https://doi.org/10.1016/j.jclinepi.2007.10.008Get rights and content

Abstract

Objectives

To determine the overall quality of reporting of meta-analyses (QUOROMs) of randomized controlled trials (RCTs) based on the QUOROM statement, to compare the reporting quality of paper-based articles and Cochrane reviews, and to determine whether compliance with the statement improves over time.

Study Design and Setting

A random sample of systematic reviews or meta-analyses of RCTs was selected from Medline (2000–2005).

Results

A total of 161 articles were included. The mean QUOROM score was 12.3 (95% confidence interval [CI]: 12.0, 12.6), which rose from 10.5 (95% CI: 8.8, 12.1) in 2000 to 13.0 (95% CI: 12.2, 13.8) in 2005. The mean QUOROM scores of Cochrane reviews and paper-based articles were 14.2 (95% CI: 13.9, 14.5) and 11.7 (95% CI: 11.3, 12.1), respectively. Compared with the paper-based articles, the Cochrane reviews had better reporting quality in the abstract section, while the quality of their trial flows was poor. The fulfillment of most QUOROM items improved with time. A linear relation of the QUOROM score with time was revealed.

Conclusions

The reporting quality of meta-analyses improves with time. The reporting quality of Cochrane reviews is better compared with paper-based articles. Room still exists for improvements in the reporting quality of both Cochrane and paper-based articles.

Introduction

The number of meta-analyses of randomized controlled trials (RCTs) has increased drastically in recent years. Meta-analysis, as a quantitative approach to combine results from similar RCTs, has earned a crucial position in providing useful information for evidence-based medicine (EBM) and health care decision making [1], [2], and the results it generates are regarded as the highest level in EBM studies [3].

Like any research enterprise, a meta-analysis can be flawed [4]. The authors of a meta-analysis report not only on the methods they used to review the articles, but also on the research methods in the articles. With this in mind, there has been an increased focus on the quality of reporting of meta-analyses (QUOROMs). A 1987 survey has discussed and explored several aspects of a meta-analysis of randomized trials including the influence of study design, control of bias, statistical analysis, and applicability of the results [5]. In 1996, the QUOROMs conference took place, where individuals with various backgrounds came together to develop guidelines for the reporting of meta-analyses. Their enormous efforts led to the publication of the QUOROM statement in 1999, which consists of a checklist and a flow diagram [6]. The checklist includes 18 items placed in six categories. It is meant to help authors improve their reporting quality and, to some extent, help with the evaluation of quality of meta-analyses.

Since its release, many journal editors and reviewers, including those involved in the Cochrane Collaboration, have pursued compliance with the QUOROM statement to provide sound, clear, and reproducible results [4], [7], [8], [9]. Several articles have evaluated the compliance with the statement [2], [10], [11], [12], [13], but the fields they focused on were limited and very small samples were given, generating results that were not always consistent with each other. In the articles, further research efforts dedicated to the influences of the QUOROM statement were called for. The overall impact of the statement on the design, conduct, and reporting of meta-analyses is yet to be unveiled.

With the exception of a few studies completed by Shea et al. [14] and Jadad et al. [15], little is known about the quality of Cochrane reviews and paper-based articles. To our knowledge, the comparison of reporting quality between Cochrane reviews and paper-based articles using the QUOROM checklist has not been evaluated nor studied.

The main purposes of this study are to determine the overall QUOROMs of RCTs based on the QUOROM statement, and to determine whether the reporting quality of meta-analyses improves with time. We also intend to compare the reporting quality of the electronic Cochrane reviews with that of the paper-based articles.

Section snippets

Search strategy and sample size

A literature search was conducted using Medline from January 2000 to December 2005 to identify meta-analyses of RCTs. The following strategy was used for search: (randomized controlled trial$ .mp. [mp = title, abstract, name of substance word, subject heading word]) AND ((systematic review OR meta analys$) .mp. [mp = title, original title, abstract, name of substance word, subject heading word]). Only articles published in English were accepted. We planned to select a random sample, a minimum

Results

Figure 1 is the flowchart for the selection of eligible articles. A total of 5,218 articles were retrieved, of which 368 were excluded for duplicates. Of the remaining 4,850 articles, 5% were selected as the study sample (n = 243) by random sampling. Of the 243 sampled articles, 82 were excluded, 48 provided no quantitative synthesis, 16 combined data from both RCTs and non-RCTs, 9 were articles of methodological research, 5 were Cochrane reviews updated in 2006, 3 were unclear whether or not the

Discussion

We often tell clinicians, health care policy makers, students, and consumers that systematic reviews represent “the best available evidence.” Unfortunately, there is relatively little empirical data on the QUOROMs or systematic reviews of RCTs. The primary purpose of this study was to determine the overall QUOROMs of RCTs. Although the results demonstrated a basically acceptable reporting quality (mean QUOROM score = 12.3), certain aspects in reporting still need improving.

In terms of complying

References (17)

There are more references available in the full text version of this article.

Cited by (62)

  • Half of Cochrane reviews were published more than 2 years after the protocol

    2020, Journal of Clinical Epidemiology
    Citation Excerpt :

    Review protocols reduce bias and ad hoc decision-making in the review process [9–11]. Although Cochrane reviews had a longer time to publication, several studies have also confirmed that they have a higher scientific quality than other systematic reviews [5,12–16], whereas one study from 2002 found no difference [17]. This consensus may be attributed to the high methodological standards that Cochrane reviews must attend to.

  • Systematic reviews in orthodontics: Impact of the PRISMA for Abstracts checklist on completeness of reporting

    2019, American Journal of Orthodontics and Dentofacial Orthopedics
    Citation Excerpt :

    One possible reason for the influence of the number of authors on the quality of reporting is the high level of precision and meticulousness required during an SR, which is perhaps more easily achieved when more authors share the responsibility and contribute to the adequate reporting of all items. In terms of the included databases, the abstracts from the Cochrane SRs achieved the best PRISMA-A scores, and it is reasonable to infer that the greater methodological rigor of Cochrane protocols may have contributed to this effect.30-32 Articles in the LILACS database returned the lowest scores, suggesting less rigor in abstract reporting; this may relate to the small number of SRs published in this database.

  • Quality of meta-analyses in major leading orthopedics journals: A systematic review

    2017, Orthopaedics and Traumatology: Surgery and Research
View all citing articles on Scopus
View full text