Published 19 October 2009, doi:10.1136/bmj.b4012
Cite this as: BMJ 2009;339:b4012

Research

Risk of bias versus quality assessment of randomised controlled trials: cross sectional study

¹ Alberta Research Centre for Health Evidence, Department of Pediatrics, University of Alberta, Aberhart Centre One, Edmonton, AB, Canada T6G 2J3

Objectives To evaluate the risk of bias tool, introduced by the Cochrane Collaboration for assessing the internal validity of randomised trials, for inter-rater agreement, concurrent validity compared with the Jadad scale and Schulz approach to allocation concealment, and the relation between risk of bias and effect estimates.

Design Cross sectional study.

Study sample 163 trials in children.

Main outcome measures Inter-rater agreement between reviewers assessing trials using the risk of bias tool (weighted ), time to apply the risk of bias tool compared with other approaches to quality assessment (paired t test), degree of correlation for overall risk compared with overall quality scores (Kendall’s statistic), and magnitude of effect estimates for studies classified as being at high, unclear, or low risk of bias (metaregression).

Results Inter-rater agreement on individual domains of the risk of bias tool ranged from slight (=0.13) to substantial (=0.74). The mean time to complete the risk of bias tool was significantly longer than for the Jadad scale and Schulz approach, individually or combined (8.8 minutes (SD 2.2) per study v 2.0 (SD 0.8), P<0.001). There was low correlation between risk of bias overall compared with the Jadad scores (P=0.395) and Schulz approach (P=0.064). Effect sizes differed between studies assessed as being at high or unclear risk of bias (0.52) compared with those at low risk (0.23).

Conclusions Inter-rater agreement varied across domains of the risk of bias tool. Generally, agreement was poorer for those items that required more judgment. There was low correlation between assessments of overall risk of bias and two common approaches to quality assessment: the Jadad scale and Schulz approach to allocation concealment. Overall risk of bias as assessed by the risk of bias tool differentiated effect estimates, with more conservative estimates for studies at low risk.

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Articles

Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study

Lesley Wood, Matthias Egger, Lise Lotte Gluud, Kenneth F Schulz, Peter Jüni, Douglas G Altman, Christian Gluud, Richard M Martin, Anthony J G Wood, and Jonathan A C Sterne
BMJ 2008 336: 601-605. [Abstract] [Full Text] [PDF]

Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors

An-Wen Chan and Douglas G Altman
BMJ 2005 330: 753. [Abstract] [Full Text] [PDF]

Measuring inconsistency in meta-analyses

Julian P T Higgins, Simon G Thompson, Jonathan J Deeks, and Douglas G Altman
BMJ 2003 327: 557-560. [Extract] [Full Text] [PDF]

Systematic reviews in health care: Assessing the quality of controlled clinical trials

Peter Jüni, Douglas G Altman, and Matthias Egger
BMJ 2001 323: 42-46. [Extract] [Full Text] [PDF]

Rapid Responses:

Read all Rapid Responses

Improving the quality of quality assessment in systematic reviews

Tobias Freund
bmj.com, 24 Oct 2009 [Full text]

Online poll

Find out more on doc2doc >>