Intended for healthcare professionals


Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study

BMJ 2008; 336 doi: (Published 13 March 2008) Cite this as: BMJ 2008;336:601

Better evidence is needed for the validity and the reliability of tools assessing the risk of bias of individual trials

In their meta-epidemiological study on the impact of inadequate concealment and blinding on estimates of effect, Wood et al. concluded that systematic reviewers should routinely assess the risk of bias in each trial[1]. Most importantly, they suggest that reviewers should report meta -analysis restricted to trials at low risk of bias. In order to make this happen, valid and reliable standardized tools are urgently needed. As a matter of fact, the reliability of different estimates of trial quality based on different methods as applied by these experts in the 3 original studies was reportedly good, but far from perfect (median k statistic was 0.67, no information provided on the range of values observed).

The use of available tools for the assessment of the quality of the trials, such as the Jadad scale [2], has been generally discouraged in the recently updated Cochrane manual[3]. The domain-based evaluation that has been proposed instead is a much welcome step forward in the right direction[2], but evidence for the reliability of its application is still needed. This is particularly relevant given the emphasis in this guidance on drawing inferences about the likely conduct of trials based on the presence or absence of items in published trial reports. Additionally, whilst all these domains have substantial face validity, only the criteria for assessing the impact of adequate/inadequate sequence generation and/or allocation concealment are empirically grounded. In particular, there is still much work to be done in relation to understanding the impact of incomplete reporting of data and other sources of potential bias on estimates of intervention effects.

1. Wood L, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG, Gluud C, Martin RM, Wood AJ, Sterne JA. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008; 336:601-605

2. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, McQuay H. Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials 1996; 17: 1-12.

3. Higgins JPT, Altman DG, eds. Assessing risk of bias in included studies. In: Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions. Version 5.0.0. Cochrane Collaboration, 2008.

Competing interests: None declared

Competing interests: No competing interests

21 March 2008
Jose M Valderas
Clinical Lecturer
Coventry PA
NIHR School of Primary Care Research. University of Manchester, Manchester M139PL