Bookmarkers beware: Bookmarks to pages other than the home page may not work after we change our server in April Editor's Choice | This Week in BMJ | Press releases BMJ No 7129 Volume 316 Letters Saturday 7 February 1998 Bias in meta-analysis detected by a simple, graphical test Graphical test is itself biased Editor, Although the concept is useful, the method proposed by Egger et al to detect bias in meta-analyses is itself biased(1): it overestimates the occurrence and extent of publication bias. This is easily shown by simulating data for a meta-analysis of a hypothetical intervention that is effective (and therefore has a negative regression coefficient by Egger et al's method) and is free of publication bias (and hence should have an intercept of zero in the regression analysis). In our simulations, each study was of a treated group and a control group, both of equal size. For each simulated meta-analysis, studies ranging from 100 per group to 1000 per group, in increments of 100, were generated. The observed number of events in each group was generated from a binomial distribution. Here is one example in which the true event rate is 40% in the control group and 10% in the treatment group. When the true population values (which would not be known in practice) are used to estimate precision, the regression coefficient is -1.7942 (an estimated log odds ratio equivalent to the expected value of 0.1667) and the intercept (0.0380, P=0.1) is close to the expected value of zero, reflecting the lack of publication bias. However, the regression coefficient estimated when the precision is based on the observed values, as would occur using Egger et al's method, is -1.7169. More importantly, the intercept is -0.4492 and significant (P0.0001), incorrectly suggesting that there has been publication bias. In general, our other simulations suggest that the bias in the estimated intercept is greater the more effective the intervention actually is and the smaller the sample size of the studies. This problem has several causes. Firstly, the estimates of precision are subject to random error due to sampling variability. This regression-dilution bias causes the regression slope to 'tilt' around the mean of the predictor and response variables so that its coefficient is closer to zero; this in turn leads to the intercept becoming negative.(2) Secondly, the estimated standardised log odds ratio is correlated with the estimated precision. Thirdly, the precision estimated by the method that we assume Egger et al used(3) is a biased estimate of the true precision, with the degree of bias increasing as sample size decreases.(4) Clearly, until the causes of the problems we have outlined are better elucidated and solutions developed, one cannot rely on the method proposed by Egger et al to detect publication bias. Les Irwig Professor of epidemiology Petra Macaskill Statistical research officer Geoffrey Berry Professor in epidemiology and biostatistics Department of Public Health and Community Medicine, A27, University of Sydney, NSW 2006, Australia Paul Glasziou Associate professor Department of Social and Preventive Medicine, University of Queensland, Medical School, Herston, QLD 4006, Australia References 1 Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997;315:629-34. (13 September.) 2 Draper N R, Smith H. Applied regression analysis. 2nd ed. New York: Wiley, 1981:122-5. 3 Galbraith R F. A note on graphical presentation of estimated odds ratios from several clinical trials. Stat Med 1988;7:889-94. 4 Agresti A. Categorical data analysis. New York: Wiley, 1990:54. Home | Current issue | Past issues | Classified ads | Career Focus | Feedback Collections | About this site | About the BMJ | BMA | Medline