Bookmarkers beware: Bookmarks to pages other than the home page may not work after we change our server in April Editor's Choice | This Week in BMJ | Press releases BMJ No 7129 Volume 316 Letters Saturday 7 February 1998 Bias in meta-analysis detected by a simple, graphical test Increase in studies of publication bias coincided with increasing use of meta-analysis Editor, Egger et al suggest a method for testing the possible existence of publication bias, based on the assumption that larger trials are more likely to be published, irrespective of their results.(1) Stern and Simes, however, suggest that large sample size is not sufficient, because of the delay in the publication of larger studies with negative results.(2) A recent letter showed that trials published at an early stage were more likely to be positive.(3) To test the association between the year of publication and treatment effect we identified 38 meta-analyses published in BMJor JAMAduring 1992-6 which provided summary data from individual studies. For each meta-analysis we tested the association between the year of publication and the treatment effect of the individual studies, using rank correlation analysis. We also tested the correlation between the sample size and the treatment effect. We ignored the sign of the correlation coefficient because it is often difficult to decide which group was the control when competing interventions were compared. Using 0.10 as a level of significance, we found that four meta-analyses showed a significant correlation between the year of publication and the treatment effect while 10 showed a significant correlation between the sample size and the treatment effect. In 25 meta-analyses the correlation coefficient between the sample size and the treatment effect was greater than that between the year of publication and the treatment effect. Therefore, both the delay to publication and the small sample size may be associated with the negative results but small sample size seems to be more important as a risk of publication bias. Publication bias jeopardises the validity of meta-analysis as well as any other attempts to use published literature. A systematic approach is crucial to identify all published studies, particularly in low circulation or non-English journals and in the grey literature, and to exclude duplicate publications of positive results.(4) We agree with Naylor that 'meta-analysis is an important contribution to research and practice but it's not a panacea.'(5) In fact, it was meta-analysis and systematic review that highlighted the problem of publication bias. By searching Medline, we found that the number of published studies (empirical, methodological, or editorial) of publication bias was 71 during 1993 to June 1997, 41 during 1987-92, three during 1981-6, and zero during 1966-80. The increase in the number of articles coincides with an increasing use of meta-analysis. It is naive to believe that publication bias did not exist or was less important a decade ago, when medical literature review was dominated by conventional non-systematic methods. Fujian Song* Senior research fellow Simon Gilbody* MRC fellow in health services research NHS Centre for Reviews and Dissemination, University of York, York YO1 5DD *The authors are undertaking a review of publication bias in systematic reviews funded by the NHS Health Technology Assessment programme. References 1 Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997;315:629-34. (13 September.) 2 Stern J M, Simes R J. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315:640-5. (13 September.) 3 Rothwell P M, Robertson G. Meta-analysis of randomised controlled trials. Lancet 1997;350:1181-2. 4 Tramer M, Reynolds D J M, Moore R A, McQuay H J. Impact of covert duplicate publication on meta-analysis: a case study. BMJ 1997;315:635-40. (13 September.) 5 Naylor C D. Meta-analysis and the meta-epidemiology of clinical research. BMJ 1997;315:617-9. (13 September.) Home | Current issue | Past issues | Classified ads | Career Focus | Feedback Collections | About this site | About the BMJ | BMA | Medline