Bookmarkers beware: Bookmarks to pages other than the home page may not work after we change our server in April Editor's Choice | This Week in BMJ | Press releases BMJ No 7129 Volume 316 Letters Saturday 7 February 1998 Bias in meta-analysis detected by a simple, graphical test Experts' views are still needed Editor, Egger et al's regression analysis of funnel plot asymmetry is an interesting exercise in descriptive statistics: most fascinating is their distribution of biasedness in meta-analyses.(1) The funnel plot test that they derive, however, rests on the assumption that it is the smaller trials that are the culprits. What if the larger trials are those that were stopped judiciously at the right moment or underwent some data-analytic 'massage'? As noted in the accompanying editorial, the predictive power of the test was validated retrospectively on eight specific instances and became positive only when its test boundaries were changed to a 10% value.(2) More experience with the test seems necessary. If we accept the test, or any similar test of heterogeneity on meta-analyses, what should we conclude from it? The main message from it is that there might be a problem because the funnel plot is asymmetrical - which we also see on the plot. The real questions to which we would like an answer are: what is the cause of the asymmetry and, more importantly, which trials should we believe? The cause of the asymmetry can be anything, from publication bias, `willingness to please' during data collection, data massage in the analysis, unclear rules for stopping the trial, or downright fraud (as indicated by Egger et al); it can also be a mix of all these things. Alternatively, the source of heterogeneity might be a true difference in underlying populations. Most difficult to live with is the overall conclusion of the test that the literature is biased. If the test is positive, should we dismiss all randomised trials on the subject? This means that we discard one trial by one group of investigators because of the results of another trial by a completely unrelated group. We might try to use quality criteria, but a recent meta-analysis on homoeopathy teaches us that this will not suffice.(3) In the end there is no escape from a return to `the expert,' who tells us which trial to believe, not only on the basis of methodology but also on the basis of insights in pathophysiology, pharmacology, and perhaps type of publication (supplements, special interest or `throw away' journals, etc). All that we can ask from the expert is a careful explanation of what arguments he or she used in accepting or dismissing the evidence from certain trials. Jan P Vandenbroucke Professor Department of Clinical Epidemiology, Leiden University Hospital, 2300 RC Leiden, Netherlands References 1 Egger M, Smith G D, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997;315:629-34. (13 September.) 2 Naylor C D. Meta-analysis and the meta-epidemiology of clinical research. BMJ 1997;315:617-9. (13 September.) 3 Vandenbroucke J P. Homeopathy trials: going nowhere. Lancet 1997;350:824. Home | Current issue | Past issues | Classified ads | Career Focus | Feedback Collections | About this site | About the BMJ | BMA | Medline