Interpretation of random effects meta-analyses

There continues to be confusion regarding the nature of meta-analysis
inference. Certain early meta-analyses (1) used what now is termed the
fixed effects model, an approach that assumes the included trials all
estimate the same underlying treatment effect. The study question is
designed to ensure that constituent trials show little important clinical
heterogeneity, and the meta-analysis estimates the overall treatment
effect. Advocates of the fixed effects model describe inference as
conditional on the trials used (2). A corollary of this model is that
considerations of clinical heterogeneity are paramount, and the
calculation of statistical heterogeneity is secondary.

Meanwhile a second approach, the random effects model, appeared (3).
It assumes trial treatment effects do not all estimate the same underlying
effect but rather that trial treatment effects are drawn from a
distribution. The calculation of the overall estimate in this approach
includes an additional term, the between-trial variance, and this approach
tends to yield larger confidence intervals than the fixed effects model.
It is asserted that the formulation of this approach allows a broader
inference - the inference applies to any future trial drawn from the
distribution. This expansive inference target was viewed with skepticism
by early commentators (4), as was the underlying conceptual construct of
the random effects model, the "universe of trials" (2). Additionally, the
random effects model suffers from an inability to reflect the large
uncertainty seen in estimating the between-trial variance term when the
number of constituent trials is small.

Riley et al (BMJ 342:d549) make a valuable addition to this
discussion. To better inform the strengths and limitations of inference,
they suggest including the 95% prediction interval which depends centrally
on the between-trial variance estimate. With this addition one has a more
valid sense of prediction for a future trial. However, on a deeper level,
this addition may only be palliative. The underlying "universe of trials"
concept remains problematic and casts a question over random effects
inferences generally. This presents a difficulty, of course, for those
relying on a broad inference from a random effects model. Clearly, the
debate between the random and fixed effects models will continue, and it
may need to be broadened to developing new models of analysis that can
quantify treatment effects across trials in the presence of substantial
clinical heterogeneity more reliably than the random effects meta-analysis
model.

Kent Johnson, MD

kent.johnson@unsw.edu.au

1. Yusuf s, Peto R, Lewis J, Collins R, Sleight P. Beta-blockage
during and after myocardial infarction: an overview of the randomized
trials. Prog Cardiovasc Dis 1985;27:335-371.

2. Peto R. Why do we need systematic overviews of randomized Trials? With
discussion.
Statist Med 1987; 6:233-244

3. DerSimonian R, Laird N. Meta-analysis in clinical trials. Controlled
Clinical Trials 1986;7:177-188.

4. Thompson S, Pocock S. Can meta-analysis be trusted? Lancet 1991;338:127
-130