Bookmarkers beware: Bookmarks to pages other than the home page may not work after we change our server in April Editor's Choice | This Week in BMJ | Press releases BMJ No 7129 Volume 316 Letters Saturday 7 February 1998 Bias in meta-analysis detected by a simple, graphical test Asymmetry detected in funnel plot was probably due to true heterogeneity Editor, Egger et al report that they 'found bias in 38% of meta-analyses published in four leading journals.'(1) This is misleading, at least insofar as our meta-analysis of inpatient geriatric consultations is concerned.(2) Firstly, the bias observed in our meta-analysis was not a retrospective detection of bias, as one might infer from Egger et al's statements. We knew that there was evidence of heterogeneity for the pooled effect estimates of geriatric consultation programmes and reported this finding.(2) Secondly, the asymmetry detected in the funnel plot of the meta-analysis of inpatient consultation programmes was probably due not to bias (distortion of true effect) but to true heterogeneity (true difference of effects between trials). We took the presence of heterogeneity as an opportunity to examine whether we could identify the programme elements that might have resulted in the observed effect differences between geriatric consultation programmes. Using a multivariate logistic regression approach, we found that both geriatric assessment programmes in which the consultant controlled the implementation of the recommendations and those that included long term follow up resulted in better outcomes than did programmes in which this was not the case. Thus, the meta-analytical methods of testing heterogeneity or drawing funnel plots should not be considered absolute criteria for separating good from bad meta-analyses. Meta-analyses reporting effect estimates that may contain bias should continue to be published in leading medical journals, as long as the possibility of heterogeneity is stated and potential underlying reasons for heterogeneity are addressed. This is especially true for meta-analyses of complex interventions. Although they are methodologically difficult to deal with, variations in effect estimates give us the opportunity to disentangle the black box of complex interventions, such as of geriatric assessment, and identify what the necessary ingredients of these programmes are.(3) A third issue concerns the 'mega-trial' to which our meta-analysis was being compared.(4) This trial was different from any of the trials considered in our meta-analysis. Among other things, it was based in a health maintenance organisation system that had incorporated considerable geriatric expertise into its usual care for older people. Another important factor was that it involved four hospital sites, each with different characteristics, populations, and survival rates. If Egger et al had taken the same pains as we did in recovering unpublished data from primary trials, they would have found that the mega-trial they used in questioning our meta-analysis was a multicentre trial with unreported variability in intervention components and outcomes across study sites. Analysts must consider rigorously any methodological issues unique to each trial, particularly when considering complex interventions. Andreas E Stuck Chief Department of Geriatrics and Rehabilitation, Zieglerspital, Berne, Switzerland Laurence Z Rubenstein Professor of geriatric medicine Education and Clinical Center, UCLA-Sepulveda VA Medical Center, Los Angeles, CA, USA Darryl Wieland Professor Department of Medicine, Division of Geriatrics, University of South Carolina School of Medicine, Columbia, SC, USA References 1 Egger M, Smith G D, Schneider M, Minder C. Bias in meta-analysis detected by a simple graphical test. BMJ 1997;315:629-34. (13 September.) 2 Stuck A E, Siu AL, Wieland G D, Adams J, Rubenstein L Z. Comprehensive geriatric assessment: a meta-analysis of controlled trials. Lancet 1993;342:1032-6. 3 Beck J C, Stuck A K. Preventing disability. Beyond the black box. JAMA 1996;276:1756-7. 4 Reuben D B, Borok G M, Wolde G T, Eshoff D H, Fishman L K, Ambrosini V E, et al. A randomised clinical trial of comprehensive geriatric assessment consultation for hospitalised HMO patients. N Engl J Med 1995;332:1345-50. Home | Current issue | Past issues | Classified ads | Career Focus | Feedback Collections | About this site | About the BMJ | BMA | Medline