Intended for healthcare professionals

Research Methods & Reporting

A guide to systematic review and meta-analysis of prediction model performance

BMJ 2017; 356 doi: https://doi.org/10.1136/bmj.i6460 (Published 05 January 2017) Cite this as: BMJ 2017;356:i6460
  1. Thomas P A Debray, assistant professor1 2,
  2. Johanna A A G Damen, PhD fellow1 2,
  3. Kym I E Snell, research fellow3,
  4. Joie Ensor, research fellow3,
  5. Lotty Hooft, associate professor1 2,
  6. Johannes B Reitsma, associate professor1 2,
  7. Richard D Riley, professor3,
  8. Karel G M Moons, professor1 2
  1. 1Cochrane Netherlands, University Medical Center Utrecht, PO Box 85500 Str 6.131, 3508 GA Utrecht, Netherlands
  2. 2Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, PO Box 85500 Str 6.131, 3508 GA Utrecht, Netherlands
  3. 3Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire, UK
  1. Correspondence to: T P A Debray T.Debray{at}umcutrecht.nl
  • Accepted 25 November 2016

Validation of prediction models is highly recommended and increasingly common in the literature. A systematic review of validation studies is therefore helpful, with meta-analysis needed to summarise the predictive performance of the model being validated across different settings and populations. This article provides guidance for researchers systematically reviewing and meta-analysing the existing evidence on a specific prediction model, discusses good practice when quantitatively summarising the predictive performance of the model across studies, and provides recommendations for interpreting meta-analysis estimates of model performance. We present key steps of the meta-analysis and illustrate each step in an example review, by summarising the discrimination and calibration performance of the EuroSCORE for predicting operative mortality in patients undergoing coronary artery bypass grafting.

Summary points

  • Systematic review of the validation studies of a prediction model might help to identify whether its predictions are sufficiently accurate across different settings and populations

  • Efforts should be made to restore missing information from validation studies and to harmonise the extracted performance statistics

  • Heterogeneity should be expected when summarising estimates of a model’s predictive performance

  • Meta-analysis should primarily be used to investigate variation across validation study results

Systematic reviews and meta-analysis are an important—if not the most important—source of information for evidence based medicine.1 Traditionally, they aim to summarise the results of publications or reports of primary treatment studies and (more recently) of primary diagnostic test accuracy studies. Compared to therapeutic intervention and diagnostic test accuracy studies, there is limited guidance on the conduct of systematic reviews and meta-analysis of primary prognosis studies.

A common aim of primary prognostic studies concerns the development of so-called prognostic prediction models or indices. These models estimate the individualised probability or risk that a certain condition will occur in the future by combining information from multiple prognostic factors from an individual. Unfortunately, there is often …

View Full Text