Bookmarkers beware: Bookmarks to pages other than the home page may not work after we change our server in April Editor's Choice | This Week in BMJ | Press releases BMJ No 7129 Volume 316 Letters Saturday 7 February 1998 Bias in meta-analysis detected by a simple, graphical test Authors' reply Editor, Bias in meta-analysis is often reflected in asymmetrical funnel plots. As we discussed in our paper, both bias and true heterogeneity in underlying effects can lead to asymmetry. Complex interventions such as geriatric consultation services may be implemented less thoroughly in larger studies, and this would explain the more positive results in smaller trials. Results of meta-analysis will then depend on how many, or how few, small or large studies are included. A thorough attempt should always be made to identify heterogeneity, and the analysis by Stuck et al is a good example of this.(1) We maintain that in these situations the combined estimate is likely to be biased and should not feature prominently in published reports. Stuck et al suggest that we should have considered differences in outcomes across centres in the health maintenance organisation trial. Post hoc analyses of effects by study centres, however, are likely to mislead, as recently shown for the |gb blocker heart attack trial.(2) Vandenbroucke could have benefited from a formal analysis of funnel plot asymmetry on at least two occasions. After visual assessment of a funnel plot he suggested that publication bias may explain the association found between passive smoking and lung cancer.(3) However, we found no evidence of asymmetry (P=0.80). Conversely, when he discussed a recent meta-analysis of homoeopathy,(4) significant funnel plot asymmetry (P0.001) would have lent support to his assertion that bias had produced a body of false positive evidence (fig).(5) Irwig et al claim that our method will overestimate the occurrence of bias. They simulated hypothetical trials of a treatment that reduced event rates from 40% to 10% (relative risk 0.25) with sample sizes ranging from 200 to 2000. Asymmetrical funnel plot of clinical trials of homoeopathy(4) (upper panel) indicating presence of bias. The linear regression of the standard normal deviate against precision (defined as the inverse of the standard error) shows a significant (P0.001) deviation of the intercept from zero (arrow). In the absence of bias, trials would scatter about a line running through the origin at standard normal deviate zero Their example is not typical of the small effects usually examined in meta-analyses. More importantly, when performing 10 000 simulations based on the same assumptions we found that on average 4.99% of tests were significant at the 5% level and 9.63% were significant at the 10% level. Therefore, contrary to Irwig et al's contention, regression dilution bias did not produce false positive results above what was expected by chance, and the P value they quote for the intercept (P0.0001), presumably based on a large number of simulations, is misleading. Seagroatt and Stratton are concerned about the specificity of our test. Considering the many possible biases, we think that the low sensitivity is of greater concern. When meta-analyses are based on a few small trials no test will be able to detect or exclude bias reliably. No statistical solution exists in this situation, and the results should be treated with great caution. Matthias Egger Reader in social medicine and epidemiology George Davey Smith Professor of clinical epidemiology Department of Social Medicine, University of Bristol, Bristol BS8 2PR Christoph Minder Head, medical statistics unit Department of Social and Preventive Medicine, University of Berne, Switzerland References 1 Stuck A K, Siu A L, Wieland G D, Adams J, Rubenstein L Z. Comprehensive geriatric assessment: a meta-analysis of controlled trials. Lancet 1993;342:1032-6. 2 Davey Smith G, Egger M. Incommunicable knowledge? Interpreting and applying the results of clinical trials and meta-analyses. J Clin Epidemiol (in press). 3 Vandenbroucke J P. Passive smoking and lung cancer: a publication bias? BMJ 1988;296:391-2. 4 Linde K, Clausius N, Ramirez G, Melchart D, Eitel F, Hedges L V, et al. Are the clinical effects of homoeopathy placebo effects? A meta-analysis of placebo-controlled trials. Lancet 1997;350:834-43. 5 Vandenbroucke J P. Homeopathy trials: going nowhere. Lancet 1997;350:824. Home | Current issue | Past issues | Classified ads | Career Focus | Feedback Collections | About this site | About the BMJ | BMA | Medline