Systematic Review
Reviews assessing the quality or the reporting of randomized controlled trials are increasing over time but raised questions about how quality is assessed

https://doi.org/10.1016/j.jclinepi.2010.04.015Get rights and content

Abstract

Objective

Many reviews specifically aimed to assess the quality of randomized controlled trials (RCTs). We evaluated the quality of reporting in such reviews.

Study Design and Setting

PubMed and the Cochrane library were searched for all reviews assessing the quality of RCTs between 1987 and 2007, and experts in the field were also contacted.

Results

We found 177 reviews published from 1987 to 2007, 58% of which were published after 2002. Of these, 131 (74%) focused on the quality of RCTs, 44 (25%) on quality of reporting, and 2 (1%) assessed both. The search strategy was well reported (92%). The criteria for assessment were reported in 97% of the reviews but were defined in only 38%. Seventy-four different items and 26 different scales were identified. Allocation sequence generation and concealment were reported in 41% and 40%, respectively, but their adequacy was assessed in 20% and 29%, respectively; scales were used in 40% and Consolidated Standards of Reporting Trials (CONSORT) checklist in 12%.

Conclusion

The number of methodological reviews has dramatically increased in recent years. Despite an improved reporting of the methodology, how quality is assessed still raises important issues. Heterogeneity of criteria used and lack of definition may limit the relevance of these reviews.

Introduction

What is new?

Key finding

  1. The number of reviews assessing the quality of randomized controlled trials (RCTs) has dramatically increased in recent years.

  2. The reporting of the methodology seem to improve over time.

  3. There is an important variability in assessing the quality of RCTs.

  4. Definition of criteria used is often lacking.

What this adds to what was known
  1. This is the first overview of all reviews assessing the quality of RCTS.

  2. Such reviews are often well reported.

  3. Nevertheless, how quality is assessed still raises important issues.

What is the implication, what should change now?
  1. To be relevant, such reviews should use well-defined and acknowledged criteria for assessing the quality of RCTs.

Randomized controlled trials (RCTs) are recognized as the gold standard to assess the efficacy of health care interventions. However, it has been demonstrated that RCTs are not necessarily unbiased [1], [2], [3]. Some studies have shown that inadequacy of certain important methodological items such as allocation concealment is associated with an exaggeration of the estimated treatment effect [1], [2], [3]. Such bias in the conduct of RCTs may have serious consequences for patients' care and decision making. This is why important concerns about the poor quality of RCTs and the low reporting of important methodological details required to judge quality have been raised, especially from the mid-1990s. These concerns led to the publication of the Consolidated Standards of Reporting Trials (CONSORT) Statement in 1996 [4] and updated in 2001 [1] that aimed to improve the reporting of RCTs by providing guidelines.

Another consequence was the increased publication of reviews that aimed to assess the quality of methods used or the reporting of RCTs. In fact, such reviews of publications have a long history. The earliest we know of was published in 1929 [5] and examined the extent to which “statistical logic” was used in a sample of 200 medical–physiological articles from current American periodicals. However, the number and the range of medical areas covered by these reviews are unknown. Also, there are no recommendations concerning their conduct, covering the search strategy, the selection of the studies to be included in the review, the process of data extraction, and the methodology of quality assessment. We aimed to retrieve and review all reviews assessing the quality or the reporting of RCTs that were published in the last 20 years to evaluate their number, characteristics, and reported methodology over time, considering especially how they assessed the RCTs.

Section snippets

Electronic search

An extensive electronic search was performed using PubMed and The Cochrane Methodology Register (issue 1, 2007) to find review articles published from January 1, 1987 to March 7, 2007. For the PubMed search, there was no MeSH term corresponding to articles assessing the quality of RCTs; thus all articles published in English between January 1, 1987 and March 7, 2007 were searched using 55 terms around the idea of quality in the title. No limits were used because some articles in PubMed may be

General characteristics of the selected studies

A flowchart of the selected methodological reviews is shown in Fig. 1. Briefly, the electronic search yielded 2,220 citations, and the search of experts' databases yielded 580 citations. From this list, 262 articles were selected for further evaluation and, finally, 177 methodological reviews were selected after reading the full text (see Appendix 2 on the journal's Web site at www.elsevier.com for the list of included references). Interrater reliability for the data extraction was very good

Discussion

To our knowledge, this study has assembled and assessed the largest database of the reviews assessing the methodological quality of RCTs in health care over 20 years. Our results showed that the number of such reviews has increased over time with variability among medical areas. These reviews, involving often recurrent authors, seem to improve over time with respect to the reporting of the search strategy and the selection of relevant references. Conversely, the criteria used to assess the

Acknowledgments

Agnes Dechartres was funded by the “Fédération Hospitalière de France” and “Assistance Publique-Hôpitaux de Paris”. Douglas Altman is funded by Cancer Research UK. The funders had no role in conducting the study or preparing the manuscript.

The authors would like to thank Dr Isabelle Boutron for reviewing the manuscript.

References (38)

  • D.G. Altman et al.

    The revised CONSORT statement for reporting randomized trials: explanation and elaboration

    Ann Intern Med

    (2001)
  • P. Jüni et al.

    Systematic reviews in health care: assessing the quality of controlled clinical trials

    BMJ

    (2001)
  • L.L. Kjaergard et al.

    Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses

    Ann Intern Med

    (2001)
  • C. Begg et al.

    Improving the quality of reporting of randomized controlled trials. The CONSORT statement

    JAMA

    (1996)
  • H.L. Dunn

    Application of statistical methods in physiology

    Physiol Rev

    (1929)
  • J.S. Bender et al.

    Quality and retrieval of obstetrical anaesthesia randomized controlled trials

    Can J Anaesth

    (1997)
  • M.J. Solomon et al.

    Randomized controlled trials in surgery

    Surgery

    (1994)
  • M. Delgado-Rodriguez et al.

    Participation of epidemiologists and/or biostatisticians and methodological quality of published controlled clinical trials

    J Epidemiol Community Health

    (2001)
  • I.H.R. Chalmers et al.

    Reporting, updating, and correcting systematic reviews of the effects of health care

    BMJ

    (1994)
  • Cited by (69)

    • Instruments assessing risk of bias of randomized trials frequently included items that are not addressing risk of bias issues

      2022, Journal of Clinical Epidemiology
      Citation Excerpt :

      Due to flaws in design and execution, the effect estimates presented in RCTs can, however, be biased. Health researchers have long acknowledged the importance of assessing “quality” of RCTs, and have developed many instruments to address this issue [3–6]. However, in 2004 the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) working group initially chose the word “quality” to represent a multidimensional concept in which risk of bias is the most critical dimension [2,5].

    • Directing scientists away from potentially biased publications: the role of systematic reviews in health care

      2021, Research Policy
      Citation Excerpt :

      Our study is unique in that we use data extracted from Cochrane reviews. The inclusion of Cochrane expert-driven assessments of bias allowed us to overcome potential issues around the identification of bias and the heterogeneities in its definition (e.g., Dechartres et al., 2011). Considering articles appraised within the same review enabled us to adopt matching criteria that are precise, and stricter than looking at the broad scientific field alone, or at the journal of publication.

    • Knowledge dissemination in clinical trials: Exploring influences of institutional support and type of innovation on selective reporting

      2018, Research Policy
      Citation Excerpt :

      Although some suggestions are provided by the analysis of prominent cases, such cases are likely to capture only the tip of the iceberg and may be of limited value for policy and prevention. Additional complications are introduced by the lack of standardised methodologies for assessing bias (e.g., Dechartres et al., 2011). Starting from the above evidence, this study attempts to generate insights into the factors associated with selective reporting.

    View all citing articles on Scopus

    Conflict of interest: none.

    View full text