Rapid Responses to:

RESEARCH: Lesley Wood, Matthias Egger, Lise Lotte Gluud, Kenneth F Schulz, Peter Jüni, Douglas G Altman, Christian Gluud, Richard M Martin, Anthony J G Wood, and Jonathan A C Sterne Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study BMJ 2008; 336: 601-605 [Abstract] [Full text]

Rapid Responses: Submit a response to this article

Rapid Responses published:

Better evidence is needed for the validity and the reliability of tools assessing the risk of bias of individual trials

Jose M Valderas, Coventry PA   (21 March 2008)

Better evidence is needed for the validity and the reliability of tools assessing the risk of bias of individual trials 21 March 2008
Jose M Valderas, Clinical Lecturer NIHR School of Primary Care Research. University of Manchester, Manchester M139PL, Coventry PA Send response to journal: Re: Better evidence is needed for the validity and the reliability of tools assessing the risk of bias of individual trials	In their meta-epidemiological study on the impact of inadequate concealment and blinding on estimates of effect, Wood et al. concluded that systematic reviewers should routinely assess the risk of bias in each trial[1]. Most importantly, they suggest that reviewers should report meta -analysis restricted to trials at low risk of bias. In order to make this happen, valid and reliable standardized tools are urgently needed. As a matter of fact, the reliability of different estimates of trial quality based on different methods as applied by these experts in the 3 original studies was reportedly good, but far from perfect (median k statistic was 0.67, no information provided on the range of values observed). The use of available tools for the assessment of the quality of the trials, such as the Jadad scale [2], has been generally discouraged in the recently updated Cochrane manual[3]. The domain-based evaluation that has been proposed instead is a much welcome step forward in the right direction[2], but evidence for the reliability of its application is still needed. This is particularly relevant given the emphasis in this guidance on drawing inferences about the likely conduct of trials based on the presence or absence of items in published trial reports. Additionally, whilst all these domains have substantial face validity, only the criteria for assessing the impact of adequate/inadequate sequence generation and/or allocation concealment are empirically grounded. In particular, there is still much work to be done in relation to understanding the impact of incomplete reporting of data and other sources of potential bias on estimates of intervention effects. 1. Wood L, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG, Gluud C, Martin RM, Wood AJ, Sterne JA. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008; 336:601-605 2. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, McQuay H. Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials 1996; 17: 1-12. 3. Higgins JPT, Altman DG, eds. Assessing risk of bias in included studies. In: Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions. Version 5.0.0. Cochrane Collaboration, 2008. www.cochrane-handbook.org/. Competing interests: None declared