Education and debate

Systematic Review: Why sources of heterogeneity in meta-analysis should be investigated

Medical Statistics Unit, London School of Hygiene and Tropical Medicine, London WC1E 7HT.

Although meta-analysis is now well established as a method of reviewing evidence, an uncritical use of the technique can be very misleading. One common problem is the failure to investigate appropriately the sources of heterogeneity, in particular the clinical differences between the studies included. This paper distinguishes between the concepts of clinical and statistical heterogeneity and exemplifies the importance of investigating heterogeneity by using published meta-analyses of epidemiological studies of serum cholesterol concentration and clinical trials of its reduction. Although not without some dangers of speculative conclusions, prompted by overzealous inspection of the data to hand, a sensible investigation of sources of heterogeneity should increase both the scientific and the clinical relevance of the results of meta-analyses.

* This paper was presented at a meeting on Systematic Reviews organised jointly by the BMJ and the UK Cochrane Centre and held in London in July 1993; it is the last . . . [Full text of this article]

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Articles

Drug eluting and bare metal stents in people with and without diabetes: collaborative network meta-analysis

Christoph Stettler, Sabin Allemann, Simon Wandel, Adnan Kastrati, Marie Claude Morice, Albert Schömig, Matthias E Pfisterer, Gregg W Stone, Martin B Leon, José Suárez de Lezo, Jean-Jacques Goy, Seung-Jung Park, Manel Sabaté, Maarten J Suttorp, Henning Kelbaek, Christian Spaulding, Maurizio Menichelli, Paul Vermeersch, Maurits T Dirksen, Pavel Cervinka, Marco De Carlo, Andrejs Erglis, Tania Chechi, Paolo Ortolani, Martin J Schalij, Peter Diem, Bernhard Meier, Stephan Windecker, and Peter Jüni
BMJ 2008 337: a1331. [Abstract] [Full Text] [PDF]

Factors predisposing women to chronic pelvic pain: systematic review

Pallavi Latthe, Luciano Mignini, Richard Gray, Robert Hills, and Khalid Khan
BMJ 2006 332: 749-755. [Abstract] [Full Text] [PDF]

This article has been cited by other articles:

• Belalcazar, L. M., Raghavan, V. A., Ballantyne, C. M. (2009). Statin-Induced Diabetes: Will It Change Clinical Practice?. Diabetes Care 32: 1941-1943 [Full text]  
• Tower, C, Ong, S S C, Ewer, A K, Khan, K, Kilby, M D (2009). Prognosis in isolated gastroschisis with bowel dilatation: a systematic review. Arch. Dis. Child. Fetal Neonatal Ed. 94: F268-F274 [Abstract] [Full text]  
• Lazo-Langner, A., Rodger, M. A., Wells, P. S. (2009). Lessons From Ximelagatran: Issues for Future Studies Evaluating New Oral Direct Thrombin Inhibitors for Venous Thromboembolism Prophylaxis in Orthopedic Surgery. CLIN APPL THROMB HEMOST 15: 316-326 [Abstract]  
• Covic, A., Kothawala, P., Bernal, M., Robbins, S., Chalian, A., Goldsmith, D. (2009). Systematic review of the evidence underlying the association between mineral metabolism disturbances and risk of all-cause mortality, cardiovascular mortality and cardiovascular events in chronic kidney disease. Nephrol Dial Transplant 24: 1506-1523 [Abstract] [Full text]  
• Bax, L., Ikeda, N., Fukui, N., Yaju, Y., Tsuruta, H., Moons, K. G. M. (2009). More Than Numbers: The Power of Graphs in Meta-Analysis. Am J Epidemiol 169: 249-255 [Abstract] [Full text]  
• Deshpande, A., Jadad, A. R. (2008). The impact of polyol-containing chewing gums on dental caries: A systematic review of original randomized controlled trials and observational studies. Journal of the American Dental Association 139: 1602-1614 [Abstract] [Full text]  
• Patsopoulos, N. A, Evangelou, E., Ioannidis, J. P. (2008). Sensitivity of between-study heterogeneity in meta-analysis: proposed metrics and empirical evaluation. Int J Epidemiol 37: 1148-1157 [Abstract] [Full text]  
• Arends, L.R., Hamza, T.H., van Houwelingen, J.C., Heijenbrok-Kal, M.H., Hunink, M.G.M., Stijnen, T. (2008). Bivariate Random Effects Meta-Analysis of ROC Curves. Med Decis Making 28: 621-638 [Abstract]  
• Stettler, C., Allemann, S., Wandel, S., Kastrati, A., Morice, M. C., Schomig, A., Pfisterer, M. E, Stone, G. W, Leon, M. B, Lezo, J. S. d., Goy, J.-J., Park, S.-J., Sabate, M., Suttorp, M. J, Kelbaek, H., Spaulding, C., Menichelli, M., Vermeersch, P., Dirksen, M. T, Cervinka, P., Carlo, M. D., Erglis, A., Chechi, T., Ortolani, P., Schalij, M. J, Diem, P., Meier, B., Windecker, S., Juni, P. (2008). Drug eluting and bare metal stents in people with and without diabetes: collaborative network meta-analysis. BMJ 337: a1331-a1331 [Abstract] [Full text]  
• Shuldham, C., Fleming, S., Yorke, J. (2008). Undertaking a systematic review: the road to successful completion. Journal of Research in Nursing 13: 282-298 [Abstract]  
• Zintzaras, E., Raman, G., Kitsios, G., Lau, J. (2008). Angiotensin-Converting Enzyme Insertion/Deletion Gene Polymorphic Variant as a Marker of Coronary Artery Disease: A Meta-analysis. Arch Intern Med 168: 1077-1089 [Abstract] [Full text]  
• Huss, A., Stuck, A. E., Rubenstein, L. Z., Egger, M., Clough-Gorr, K. M. (2008). Multidimensional Geriatric Assessment: Back to the Future Multidimensional Preventive Home Visit Programs for Community-Dwelling Older Adults: A Systematic Review and Meta-Analysis of Randomized Controlled Trials. Journals of Gerontology Series A: Biological Sciences and Medical Sciences 63: 298-307 [Abstract] [Full text]  
• Garg, A. X., Hackam, D., Tonelli, M. (2008). Systematic Review and Meta-analysis: When One Study Is Just not Enough. CJASN 3: 253-260 [Full text]  
• Riley, R. D., Thompson, J. R., Abrams, K. R. (2008). An alternative model for bivariate random-effects meta-analysis when the within-study correlations are unknown. Biostatistics 9: 172-186 [Abstract] [Full text]  
• Trelle, S., Reichenbach, S., Juni, P. (2007). Chondroitin for Osteoarthritis of the Knee or Hip. ANN INTERN MED 147: 884-885 [Full text]  
• Mantelli, F, Santos, M S, Petitti, T, Sgrulletta, R, Cortes, M, Lambiase, A, Bonini, S (2007). Systematic review and meta-analysis of randomised clinical trials on topical treatments for vernal keratoconjunctivitis. Br J Ophthalmol 91: 1656-1661 [Abstract] [Full text]  
• Vanhoenacker, P. K., Heijenbrok-Kal, M. H., Van Heste, R., Decramer, I., Van Hoe, L. R., Wijns, W., Hunink, M. G. M. (2007). Diagnostic Performance of Multidetector CT Angiography for Assessment of Coronary Artery Disease: Meta-analysis. Radiology 244: 419-428 [Abstract] [Full text]  
• Reichenbach, S., Sterchi, R., Scherer, M., Trelle, S., Burgi, E., Burgi, U., Dieppe, P. A., Juni, P. (2007). Meta-analysis: Chondroitin for Osteoarthritis of the Knee or Hip. ANN INTERN MED 146: 580-590 [Abstract] [Full text]  
• Collard, H. R. (2007). On Beyond Zebra: Understanding Rare Diseases. Chest 131: 644-646 [Full text]  
• Hatton, R. C, Winterstein, A. G, McKelvey, R. P, Shuster, J., Hendeles, L. (2007). Efficacy and Safety of Oral Phenylephrine: Systematic Review and Meta-Analysis. The Annals of Pharmacotherapy 41: 381-390 [Abstract] [Full text]  
• Zintzaras, E., Kaditis, A. G. (2007). Sleep-Disordered Breathing and Blood Pressure in Children: A Meta-analysis. Arch Pediatr Adolesc Med 161: 172-178 [Abstract] [Full text]  
• The Fibrinogen Studies Collaboration, (2006). Regression dilution methods for meta-analysis: assessing long-term variability in plasma fibrinogen among 27 247 adults in 15 prospective studies. Int J Epidemiol 35: 1570-1578 [Abstract] [Full text]  
• Thakkinstian, A., Han, P., McEvoy, M., Smith, W., Hoh, J., Magnusson, K., Zhang, K., Attia, J. (2006). Systematic review and meta-analysis of the association between complementary factor H Y402H polymorphisms and age-related macular degeneration. Hum Mol Genet 15: 2784-2790 [Abstract] [Full text]  
• Geddes, J. (2006). Meta-analysis in the assessment of treatment outcome. J Psychopharmacol 20: 67-71 [Abstract]  
• Latthe, P., Mignini, L., Gray, R., Hills, R., Khan, K. (2006). Factors predisposing women to chronic pelvic pain: systematic review. BMJ 332: 749-755 [Abstract] [Full text]  
• Bagshaw, S. M., McAlister, F. A., Manns, B. J., Ghali, W. A. (2006). Acetylcysteine in the Prevention of Contrast-Induced Nephropathy: A Case Study of the Pitfalls in the Evolution of Evidence. Arch Intern Med 166: 161-166 [Abstract] [Full text]  
• Ades, A. E., Lu, G., Higgins, J. P. T. (2005). The Interpretation of Random-Effects Meta-Analysis in Decision Models. Med Decis Making 25: 646-654 [Abstract]  
• White, I. R (2005). Uses and limitations of randomization-based efficacy estimators. Stat Methods Med Res 14: 327-347 [Abstract]  
• Thakkinstian, A., McEvoy, M., Minelli, C., Gibson, P., Hancox, B., Duffy, D., Thompson, J., Hall, I., Kaufman, J., Leung, T.-f., Helms, P. J., Hakonarson, H., Halpi, E., Navon, R., Attia, J. (2005). Systematic Review and Meta-Analysis of the Association between {beta}2-Adrenoceptor Polymorphisms and Asthma: A HuGE Review. Am J Epidemiol 162: 201-211 [Abstract] [Full text]  
• Troosters, T., Casaburi, R., Gosselink, R., Decramer, M. (2005). Pulmonary Rehabilitation in Chronic Obstructive Pulmonary Disease. Am. J. Respir. Crit. Care Med. 172: 19-38 [Full text]  
• Bagshaw, S. M., Ghali, W. A. (2005). Theophylline for Prevention of Contrast-Induced Nephropathy: A Systematic Review and Meta-analysis. Arch Intern Med 165: 1087-1093 [Abstract] [Full text]  
• Cho, H. J., Hotopf, M., Wessely, S. (2005). The Placebo Response in the Treatment of Chronic Fatigue Syndrome: A Systematic Review and Meta-Analysis. Psychosom. Med. 67: 301-313 [Abstract] [Full text]  
• Deans, K. J., Natanson, C., Minneci, P. C. (2004). Dose Effects of Steroids on Survival in Sepsis. ANN INTERN MED 141: 892-892 [Full text]  
• Lee, V. C., Rhew, D. C., Dylan, M., Badamgarav, E., Braunstein, G. D., Weingarten, S. R. (2004). Meta-Analysis: Angiotensin-Receptor Blockers in Chronic Heart Failure and High-Risk Acute Myocardial Infarction. ANN INTERN MED 141: 693-704 [Abstract] [Full text]  
• Goodwin, G. M., Geddes, J. R. (2004). Introduction to Systematic Reviews. J Psychopharmacol 18: 249-250  
• Arterburn, D. E., Crane, P. K., Veenstra, D. L. (2004). The Efficacy and Safety of Sibutramine for Weight Loss: A Systematic Review. Arch Intern Med 164: 994-1003 [Abstract] [Full text]  
• Casey, J. R., Pichichero, M. E. (2004). Meta-analysis of Cephalosporin Versus Penicillin Treatment of Group A Streptococcal Tonsillopharyngitis in Children. Pediatrics 113: 866-882 [Abstract] [Full text]  
• Ades, A. E., Lu, G., Claxton, K. (2004). Expected Value of Sample Information Calculations in Medical Decision Modeling. Med Decis Making 24: 207-227 [Abstract]  
• Quinlan, D. J., McQuillan, A., Eikelboom, J. W. (2004). Low-Molecular-Weight Heparin Compared with Intravenous Unfractionated Heparin for Treatment of Pulmonary Embolism: A Meta-Analysis of Randomized, Controlled Trials. ANN INTERN MED 140: 175-183 [Abstract] [Full text]  
• van Dijk, M. A.J., Reitsma, J. B., Fischer, J. C., Sanders, G. T.B. (2003). Indications for Requesting Laboratory Tests for Concurrent Diseases in Patients with Carpal Tunnel Syndrome: A Systematic Review. Clin. Chem. 49: 1437-1444 [Abstract] [Full text]  
• Grimshaw, J, McAuley, L M, Bero, L A, Grilli, R, Oxman, A D, Ramsay, C, Vale, L, Zwarenstein, M (2003). Systematic reviews of the effectiveness of quality improvement strategies and programmes. Qual Saf Health Care 12: 298-303 [Abstract] [Full text]  
• Bennett, D. A (2003). Review of analytical methods for prospective cohort studies using time to event data: single studies and implications for meta-analysis. Stat Methods Med Res 12: 297-319 [Abstract]  
• van der Windt, D A W M, Bouter, L M (2003). Physiotherapy or corticosteroid injection for shoulder pain?. Ann Rheum Dis 62: 385-387 [Full text]  
• Owen, C G, Fletcher, A E, Donoghue, M, Rudnicka, A R (2003). How big is the burden of visual loss caused by age related macular degeneration in the United Kingdom?. Br J Ophthalmol 87: 312-317 [Abstract] [Full text]  
• Lorant, V., Deliege, D., Eaton, W., Robert, A., Philippot, P., Ansseau, M. (2003). Socioeconomic Inequalities in Depression: A Meta-Analysis. Am J Epidemiol 157: 98-112 [Abstract] [Full text]  
• Cooper, N J, Sutton, A J, Abrams, K R (2002). Decision analytical economic modelling within a Bayesian framework: application to prophylactic antibiotics use for caesarean section. Stat Methods Med Res 11: 491-512 [Abstract]  
• Schulz, K. F., Altman, D. G., Moher, D., Juni, P., Egger, M., Balk, E. M., Bonis, P. A. L., Lau, J., Ioannidis, J. P. A. (2002). Allocation Concealment in Clinical Trials. JAMA 288: 2406-2409 [Full text]  
• Wheatley, K., Ives, N., Hancock, B., Gore, M., Lens, M.B., Dawes, M. (2002). Need for a Quantitative Meta-Analysis of Trials of Adjuvant Interferon in Melanoma. JCO 20: 4120-4122 [Full text]  
• Mastrangelo, G., Fedeli, U., Fadda, E., Milan, G., Lange, J. H (2002). Epidemiologic evidence of cancer risk in textile industry workers: a review and update. Toxicol Ind Health 18: 171-181 [Abstract]  
• Jordan, R., Gold, L., Cummins, C., Hyde, C. (2002). Systematic review and meta-analysis of evidence for increasing numbers of drugs in antiretroviral combination therapy. BMJ 324: 757-757 [Abstract] [Full text]  
• Stuck, A. E., Egger, M., Hammer, A., Minder, C. E., Beck, J. C. (2002). Home Visits to Prevent Nursing Home Admission and Functional Decline in Elderly People: Systematic Review and Meta-regression Analysis. JAMA 287: 1022-1028 [Abstract] [Full text]  
• Dickersin, K. (2002). Systematic reviews in epidemiology: why are we so far behind?. Int J Epidemiol 31: 6-12 [Full text]  
• Higgins, J. P., Spiegelhalter, D. J (2002). Being sceptical about meta-analyses: a Bayesian perspective on magnesium trials in myocardial infarction. Int J Epidemiol 31: 96-104 [Abstract] [Full text]  
• Thompson, S. G, Turner, R. M, Warn, D. E (2001). Multilevel models for meta-analysis, and their application to absolute risk differences. Stat Methods Med Res 10: 375-392 [Abstract]  
• Torgerson, D. J., Bell-Syer, S. E. M. (2001). Hormone Replacement Therapy and Prevention of Nonvertebral Fractures: A Meta-analysis of Randomized Trials. JAMA 285: 2891-2897 [Abstract] [Full text]  
• Song, F., Sheldon, T. A., Sutton, A. J., Abrams, K. R., Jones, D. R. (2001). Methods for Exploring Heterogeneity in Meta-Analysis. Eval Health Prof 24: 126-151 [Abstract]  
• Shibata, M. C., Flather, M. D., Wang, D. (2001). Systematic review of the impact of beta blockers on mortality and hospital admissions in heart failure. Eur J Heart Fail 3: 351-357 [Abstract] [Full text]  
• Prior, C., Clements, J., Rowett, M., Taylor, D., Rowsell, R., Link, C., Donoghue, J., Lee, S., Anderson, I., Duggins, R., Rhinds, D., Hall, W., Kerwin, R., Adams, C., Duggan, L., Geddes, J., Harrison, P., Freemantle, N., Bebbington, P. (2001). Atypical antipsychotics in the treatment of schizophrenia. BMJ 322: 924a-924 [Full text]  
• WEST, R R (2000). Evidence based medicine overviews, bulletins, guidelines, and the new consensus. Postgrad. Med. J. 76: 383-389 [Full text]  
• Weed, D. L (2000). Interpreting epidemiological evidence: how meta-analysis and causal inference methods are related. Int J Epidemiol 29: 387-390 [Abstract] [Full text]  
• Kong, D. F., Topol, E. J., Bittl, J. A., White, H. D., Theroux, P., Hasselblad, V., Califf, R. M. (1999). Clinical Outcomes of Bivalirudin for Ischemic Heart Disease. Circulation 100: 2049-2053 [Abstract] [Full text]  
• DerSimonian, R., Levine, R. J. (1999). Resolving Discrepancies Between a Meta-analysis and a Subsequent Large Controlled Trial. JAMA 282: 664-670 [Abstract] [Full text]  
• Lemmer, B., Grellier, R., Steven, J. (1999). Systematic Review of Nonrandom and Qualitative Research Literature: Exploring and Uncovering an Evidence Base for Health Visiting and Decision Making. Qual Health Res 9: 315-328 [Abstract]  
• Kong, D. F., Califf, R. M., Miller, D. P., Moliterno, D. J., White, H. D., Harrington, R. A., Tcheng, J. E., Lincoff, A. M., Hasselblad, V., Topol, E. J. (1998). Clinical Outcomes of Therapeutic Agents That Block the Platelet Glycoprotein IIb/IIIa Integrin in Ischemic Heart Disease. Circulation 98: 2829-2835 [Abstract] [Full text]  
• Moher, D., Jadad, A. R., Klassen, T. P. (1998). Guides for Reading and Interpreting Systematic Reviews: III. How Did the Authors Synthesize the Data and Make Their Conclusions?. Arch Pediatr Adolesc Med 152: 915-920 [Full text]  
• Griffin, S., Greenhalgh, T. (1998). Diabetes care in general practice: meta-analysis of randomised control trials • Commentary: Meta-analysis is a blunt and potentially misleading instrument for analysing models of service delivery. BMJ 317: 390-396 [Abstract] [Full text]  
• Kozyrskyj, A. L., Hildes-Ripstein, G. E., Longstaffe, S. E. A., Wincott, J. L., Sitar, D. S., Klassen, T. P., Moffatt, M. E. K. (1998). Treatment of Acute Otitis Media With a Shortened Course of Antibiotics: A Meta-analysis. JAMA 279: 1736-1742 [Abstract] [Full text]  
• Harrington, R., Whittaker, J., Shoebridge, P., Campbell, F. (1998). Systematic review of efficacy of cognitive behaviour therapies in childhood and adolescent depressive disorder. BMJ 316: 1559-1563 [Abstract] [Full text]  
• Tang, J L, Armitage, J M, Lancaster, T, Silagy, C A, Fowler, G H, Neil, H A W, Smith, G. D., Ebrahim, S. (1998). Systematic review of dietary intervention trials to lower blood total cholesterol in free-living subjects • Commentary: Dietary change, cholesterol reduction, and the public health---what does meta-analysis add?. BMJ 316: 1213-1220 [Abstract] [Full text]  
• Schierhout, G., Roberts, I. (1998). Prophylactic antiepileptic agents after head injury: a systematic review. J. Neurol. Neurosurg. Psychiatry 64: 108-112 [Abstract] [Full text]  
• Fujian Song, (1997). Risperidone in the treatment of schizophrenia: a meta-analysis of randomized controlled trials. J Psychopharmacol 11: 65-71 [Abstract]  
• Kellner, J. D., Ohlsson, A., Gadomski, A. M., Wang, E. E. L. (1996). Efficacy of Bronchodilator Therapy in Bronchiolitis: A Meta-analysis. Arch Pediatr Adolesc Med 150: 1166-1172 [Abstract]  
• Sharp, S. J, Thompson, S. G, Altman, D. G (1996). The relation between treatment benefit and underlying risk in meta-analysis. BMJ 313: 735-738 [Full text]  
• Moher, D., Olkin, I. (1995). Meta-analysis of Randomized Controlled Trials: A Concern for Standards. JAMA 274: 1962-1964 [Abstract]  
• Jeng, G. T., Scott, J. R., Burmeister, L. F. (1995). A Comparison of Meta-analytic Results Using Literature vs Individual Patient Data: Paternal Cell Immunization for Recurrent Miscarriage. JAMA 274: 830-836 [Abstract]  
• (1995). Magnesium, myocardial infarction, meta-analysis and megatrials. DTB 33: 25-27 [Abstract] [Full text]