Bias in meta-analysis detected by a simple, graphical test

doi:10.1136/bmj.316.7129.469

From trainee to consultant, BMJ Group offers doctors around the world tailored information, special events, learning resources and recruitment services at every step along their career path.

... by doctors, for doctors, for patients
We are open for entries!

Online learning

BMJ Learning

High-quality CME / CPD for doctors and other healthcare professionals. BMJ Learning features hundreds of accredited, peer-reviewed learning modules in text, video, and audio formats. Find out more

Courses and Qualifications

BMJ Masterclasses

BMJ Masterclasses, led by experts, help clinicians to use the latest evidence and recent guidelines in practice and meet their CPD/CME requirements. Find out more

Exam Preparation

The leading provider of online exam preparation, helping over 167,000 healthcare professionals to pass their exams. Find out more

onExamination

Search all BMJ comment articles:

From

Limit by

Access to the full text of this article requires a subscription or payment. Please log in or subscribe below.

Asymmetry detected in funnel plot was probably due to true heterogeneity

Andreas E Stuck, Chiefa,
Laurence Z Rubenstein, Professor of geriatric medicineb,
Darryl Wieland, Professorc

^a Department of Geriatrics and Rehabilitation, Zieglerspital, Berne, Switzerland
^b Education and Clinical Center, UCLA-Sepulveda VA Medical Center, Los Angeles, CA, USA
^c Department of Medicine, Division of Geriatrics, University of South Carolina School of Medicine, Columbia, SC, USA
^d Department of Clinical Epidemiology, Leiden University Hospital, 2300 RC Leiden, Netherlands
^e Department of Public Health and Community Medicine, A27, University of Sydney, NSW 2006, Australia
^f Department of Social and Preventive Medicine, University of Queensland, Medical School, Herston, QLD 4006, Australia
^g Unit of Healthcare Epidemiology, Institute of Health Sciences, Oxford University, Oxford OX3 7LF
^h Diabetes Research Laboratories, Radcliffe Infirmary, Oxford OX2 6HE
ⁱ Department of Social Medicine, University of Bristol, Bristol BS8 2PR
^j Department of Social and Preventive Medicine, University of Berne, Switzerland
^k Academic Section of Geriatric Medicine, Royal Infirmary, Glasgow G4 0SF
^l NHS Centre for Reviews and Dissemination, University of York, York YO1 5DD

Editor—Egger et al report that they “found bias in 38% of meta-analyses published in four leading journals.”1 This is misleading, at least insofar as our meta-analysis of inpatient geriatric consultations is concerned.2

Firstly, the bias observed in our meta-analysis was not a retrospective detection of bias, as one might infer from Egger et al's statements. We knew that there was evidence of heterogeneity for the pooled effect estimates of geriatric consultation programmes and reported this finding.2 Secondly, the asymmetry detected in the funnel plot of the meta-analysis of inpatient consultation programmes was probably due not to bias (distortion of true effect) but to true heterogeneity (true difference of effects between trials). We took the presence of heterogeneity as an opportunity to examine whether we could identify the programme elements that might have resulted in the observed effect differences between geriatric consultation programmes. Using a multivariate logistic regression approach, we found that both geriatric assessment programmes in which the consultant controlled the implementation of the recommendations and those that included long term follow up resulted in better outcomes than did programmes in which this was not the case.

Thus, the meta-analytical methods of testing heterogeneity or drawing funnel plots should not be considered absolute criteria for separating good from bad meta-analyses. Meta-analyses reporting effect estimates that may contain bias should continue to be published in leading medical journals, as long as the possibility of heterogeneity is stated and potential underlying reasons for heterogeneity are addressed. This is especially true for meta-analyses of complex interventions. Although they are methodologically difficult to deal with, variations in effect estimates give us the opportunity to disentangle the black box of complex interventions, such as of geriatric assessment, and identify what the necessary ingredients of these programmes are.3

A third issue concerns …