Meta-analysis and the meta-epidemiology of clinical research

Meta-analysis is an important contribution to research and practice but it's not a panacea

This week's BMJ contains a pot-pourri of materials that deal with the research methodology of meta-analysis. Meta-analysis in clinical research is based on simple principles: systematically searching out, and, when possible, quantitatively combining the results of all studies that have addressed a similar research question. Given the information explosion in clinical research, the logic of basing research reviews on systematic searching and careful quantitative compilation of study results is incontrovertible. However, one aspect of meta-analysis as applied to randomised trials has always been controversial1 2 –combining data from multiple studies into single estimates of treatment effect.

In theory, aggregation of data from multiple trials should enhance the precision and accuracy of any pooled result. But combining data requires a leap of faith: it presumes that the differences among studies are primarily due to chance. In fact, differences in the direction or size of treatment effects may be caused by other factors, including subtle differences in treatments, populations, outcome measures, study design, and study quality.3 Thus meta-analyses may generate misleading results by ignoring meaningful heterogeneity among studies, entrenching the biases in individual studies, and introducing further biases through the process of finding studies and selecting results to be pooled.

Our understanding of these limits of meta-analysis has arisen partly because a generation of investigators has stepped back from the unthinking pooling of data and begun researching clinical research itself. Those interested in the science of systematic reviews focus on trials as the unit of analysis; and along the way they have usefully shifted the goalposts for reporting on clinical research.