Intended for healthcare professionals

  1. Research
  2. Transparent reporting...
  3. Transparent reporting of multivariable prediction models developed or validated using clustered data: TRIPOD-Cluster checklist
CCBY Open access
Research Methods & Reporting

Transparent reporting of multivariable prediction models developed or validated using clustered data: TRIPOD-Cluster checklist

BMJ 2023; 380 doi: https://doi.org/10.1136/bmj-2022-071018 (Published 07 February 2023) Cite this as: BMJ 2023;380:e071018

Linked RMR

Transparent reporting of multivariable prediction models developed or validated using clustered data
  1. Thomas P A Debray, assistant professor1 2,
  2. Gary S Collins, professor of medical statistics3 4,
  3. Richard D Riley, professor of biostatistics5,
  4. Kym I E Snell, senior lecturer in biostatistics5,
  5. Ben Van Calster, associate professor6,
  6. Johannes B Reitsma, associate professor1 2,
  7. Karel G M Moons, professor of clinical epidemiology 1 2
  1. 1Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
  2. 2Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
  3. 3Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Botnar Research Centre, University of Oxford, Oxford, UK
  4. 4NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford, UK
  5. 5Centre for Prognosis Research, School of Primary, Community and Social Care, Keele University, Keele, UK
  6. 6Department of Development and Regeneration, KU Leuven, Leuven, Belgium
  1. Correspondence to: T Debray T.Debray{at}umcutrecht.nl
  • Accepted 9 August 2022

The increasing availability of large combined datasets (or big data), such as those from electronic health records and from individual participant data meta-analyses, provides new opportunities and challenges for researchers developing and validating (including updating) prediction models. These datasets typically include individuals from multiple clusters (such as multiple centres, geographical locations, or different studies). Accounting for clustering is important to avoid misleading conclusions and enables researchers to explore heterogeneity in prediction model performance across multiple centres, regions, or countries, to better tailor or match them to these different clusters, and thus to develop prediction models that are more generalisable. However, this requires prediction model researchers to adopt more specific design, analysis, and reporting methods than standard prediction model studies that do not have any inherent substantial clustering. Therefore, prediction model studies based on clustered data need to be reported differently so that readers can appraise the study methods and findings, further increasing the use and implementation of such prediction models developed or validated from clustered datasets.

The TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) statement provides guidance for the reporting of studies developing, validating, or updating a multivariable prediction model. However, the TRIPOD statement does not fully cover the specific methodological and analytical complexity in prediction model studies that are based on (and thus should account for) clustered data. We therefore developed TRIPOD-Cluster, a new reporting checklist for prediction model studies that are based on clustered datasets. Ten new items are introduced and 19 of the original TRIPOD items have been updated to better reflect issues for clustering. The rationale for each item is discussed, along with examples of good reporting and why transparent reporting is important. TRIPOD-Cluster is best used in conjunction with the TRIPOD-Cluster explanation and elaboration document (www.tripod-statement.org).

Prediction models combine a number of characteristics (predictors or covariates) to produce an individual’s probability or risk of currently having (diagnosis) or developing in a certain time period (prognosis) a specific condition or outcome.12345 They are (still) typically derived using multivariable regression techniques,26 such as logistic regression (for diagnostic and short term binary prognostic outcomes), Cox proportional hazards regression (for long term binary outcomes), linear regression (for continuous outcomes), or multinomial regression (for categorical outcomes). However, they can be and, increasingly, are developed using machine learning algorithms such as neural networks, support vector machines, and random forests.7

Prediction models are widely used to support medical practice and diagnostic, prognostic, therapeutic, and preventive decision making.34 Therefore, prediction models should be developed, validated, and updated or tailored to different settings using appropriate methods and data sources, such that their predictions are sufficiently accurate and supportive for further decision making in the targeted population and setting.1289 To evaluate whether a prediction model is fit for purpose, to properly assess their quality and any risks of bias, full and transparent reporting of prediction model studies is essential.10

The TRIPOD statement was published in 2015 to provide guidance on the reporting of prediction model studies.1112 The TRIPOD statement comprises a checklist of 22 minimum reporting items that should be addressed in all prediction model studies, with translations available in Chinese and Japanese. However, the TRIPOD statement does not cover important items that arise when prediction models are based on clustered datasets.1314 These datasets arise when individual participant data are obtained or combined from multiple groups, centres, settings, regions, countries, or studies (referred to here as clusters). Observations within a cluster tend to be more alike than observations from other clusters, leading to cluster populations that are different but related.

Clustered datasets can, for instance, be obtained by combining individual participant data from multiple studies (clustering by study; table 1), by conducting multicentre studies (clustering by centre), or by retrieving individual participant data from registries or datasets with electronic healthcare records (clustering by healthcare centre or geographical region; table 2). If differences between clusters are left unresolved during prediction model development, they might not perform well when applied in other or new clusters, and therefore have limited generalisability.2930 Similarly, when heterogeneity between clusters is ignored during prediction model validation, estimates of prediction model performance can be highly misleading.313233 In clustered or combined datasets, different individuals in the same cluster have been subject to, for example, similar healthcare processes that have also been delivered by the same healthcare providers, and therefore could be more alike than individuals from different clusters. Sometimes, clusters might also differ in participant eligibility criteria or participation, follow-up length, predictor and outcome definitions, and in (the quality of) applied measurement methods. Hence correlation is likely to be present between observations from the same data cluster,29 which can lead to differences or heterogeneity between clusters regarding patient characteristics, baseline risk, predictor effects, and outcome occurrence.

Table 1

Examples of prediction model studies that are based on individual participant data meta-analysis from various medical domains and settings

View this table:
Table 2

Examples of prediction model studies that are based on electronic healthcare records from various medical domains and settings

View this table:

The presence of clustering has important benefits for prediction model research because it enables us to explore heterogeneity in prediction model performance (eg, the model’s calibration and discrimination) across, for example, multiple subgroups, centres, regions, or countries. Identifying sources of heterogeneity helps to better tailor or match the models to these different clusters, and thus to develop prediction models that are more generalisable. However, this strategy requires prediction model researchers to adopt more specific methods for design, analysis, and reporting than standard prediction model studies that do not have any inherent substantial clustering. Therefore, prediction model studies based on clustered data need to be reported differently, so that readers can appraise the study methods and findings, further increasing the use and implementation of prediction models developed or validated from clustered datasets. These specifics are not explicitly mentioned in the original TRIPOD statement.133031

Similar to how CONSORT Cluster is an extension of CONSORT,32 we developed a new standalone prediction model reporting guideline, TRIPOD-Cluster, that provides guidance for transparent reporting of studies that describe the development or validation (including updating) of a diagnostic or prognostic prediction model using clustered data. TRIPOD-Cluster focuses on clustered studies as present in individual participant data meta-analysis (IPD-MA) and electronic healthcare records datasets (table 1 and table 2), and not on studies with clustering defined by repeated measurements within the same individuals. TRIPOD-Cluster is best used in conjunction with the TRIPOD-Cluster explanation and elaboration document. To aid the editorial process and readability of prediction model studies based on clustered data, it is recommended that authors include a completed checklist in their submission. All information is available online (www.tripod-statement.org).

Summary points

  • To evaluate whether a prediction model is fit for purpose, and to properly assess their quality and any risks of bias, full and transparent reporting of prediction model studies is essential

  • The TRIPOD-Cluster statement is a new reporting checklist for prediction model studies that are based on clustered datasets

  • Clustered datasets can be obtained by combining individual participant data from multiple studies, by conducting multicentre studies, or by retrieving individual participant data from registries or datasets with electronic healthcare records

  • Presence of clustering can lead to differences (or heterogeneity) between clusters regarding participant characteristics, baseline risk, predictor effects, and outcome occurrence

  • Performance of prediction models can vary across clusters, and thereby affect their generalisability

  • Additional reporting efforts are needed in clustered data to clarify the identification of eligible data sources, data preparation, risk-of-bias assessment, heterogeneity in prediction model parameters, and heterogeneity in prediction model performance estimates

Predictions with clustered data: challenges and opportunities

Overfitting is a common problem in prediction model studies that use a single and often small dataset from a specific setting (eg, a single hospital or practice). Prediction models that are overfitted or overly simple tend to have poor accuracy when applied to new individuals from other hospitals or practices. In addition, when validation studies are also based on single, small, and local datasets, estimates of a prediction model’s performance likely become inaccurate and too optimistic, and fail to show generalisability across multiple medical practices, settings, or countries.33 Many examples of prediction models do not perform adequately when applied to new patients.343536

To overcome these problems, interest in prediction model studies using large clustered datasets has grown. Combining data from multiple clusters has various advantages:

  • Prediction models developed, validated, or updated from larger combined datasets are less prone to overfitting, and estimates of prediction model performance are less likely to be optimistic.

  • Researchers can better investigate more complex but more predictive associations, such as including non-linearity of predictor effects, predictor interactions, and time varying effects.

  • The presence of multiple large clusters allows development and direct validation of models across a wide range of populations and settings, and thus allows direct tests of a model’s generalisability.1337

  • Researchers can better explore and quantify any heterogeneity in a model’s predictive performance across multiple subgroups, centres, regions, or countries to better tailor or match the models to these different clusters, which is desirable when deciding on their use for individualised decision support.38

All these benefits of prediction modelling using clustered datasets improve model performance across new clusters, and help develop, validate, and update prediction models that are more generalisable across multiple types of clusters. Prediction model studies using clustered data include important additional details that are not explicitly detailed in the original TRIPOD statement.1112 These details include varying predictor and outcome definitions across clusters, varying (quality of) predictor and outcome measurements, varying inclusion criteria or participation of individuals, the presence and handling of systematically missing data across and within clusters (ie, a predictor not measured for any individuals in a cluster), and varying design related aspects such as blinding. Accordingly, when a clustered dataset is used, special efforts in the development and validation of the prediction model might be needed during the analysis to avoid bias.

In general, adopting statistical methods that specifically account for clustering is recommended, because the presence of cluster effects could lead to differences in estimated model parameters and in the model’s predictive accuracy across clusters, such that ignoring clustering leads to limited generalisability and applicability.39404142 For instance, random effects models might be needed during prediction model development to allow for heterogeneity across clusters in baseline risk and predictor effects, which yields prediction models with varying intercepts and predictor effects for different clusters.3940 Random effects models might also be needed during model validation to assess heterogeneity in prediction model performance across clusters to gain a deeper understanding of where and in which clusters the model does not satisfy.37 Other strategies to account for clustering might consider adjustment for cluster level variables, or implement variable selection algorithms that reduce the presence of cluster heterogeneity in model predictions.41 Strategies to explore the presence of clustering and account for statistical heterogeneity across clusters should be adequately reported, and are discussed in the TRIPOD-Cluster explanation and elaboration document.

None of the aspects mentioned above is included in the original TRIPOD checklist. For this reason, the present TRIPOD-Cluster checklist provides guidance for the reporting of studies describing the development, validation, and updating of a multivariable diagnostic or prognostic prediction model using clustered data. We provide a flow chart to help authors decide whether to use TRIPOD-Cluster or the original TRIPOD checklist (fig 1). We further refer to the explanation and elaboration document for detailed reasoning and guidance why a new cluster specific item has been added to the original TRIPOD checklist, in which we also use the example studies listed in table 1 and table 2.

Fig 1
Fig 1

Flow chart to determine whether the published study should follow the original TRIPOD statement or TRIPOD-Cluster checklist

Development of TRIPOD-Cluster

We followed the strategy proposed by the EQUATOR (enhancing the quality and transparency of health research) Network (www.equator-network.org) for developing health research reporting guidelines and formed an executive group (consisting of Doug Altman, GC, TD, KM, JR, and RR).42

The first step consisted of reviewing the literature and the EQUATOR Network database (www.equator-network.org) to identify existing reporting guidelines for large (clustered) data or IPD-MA studies in general, including any extensions of these guidelines. Reporting guidelines have been published for various research designs, and these contain many items that are also relevant to prediction model studies using clustered data. In particular, the PRISMA (preferred reporting items for systematic reviews and meta-analyses) checklist of individual participant data (PRISMA-IPD),43 STROBE (strengthening the reporting of obeservational studies in epidemiology) statement,44 and RECORD (reporting of studies conducted using observational routinely collected data) guideline45 contain several items that are also relevant for the transparent reporting of prediction model studies using datasets from electronic healthcare records, multicentre cohort studies, or IPD-MA studies.

The executive group subsequently convened 11 times in person over four years to discuss the development and structure of TRIPOD-Cluster, the rationale for including new items, topics requiring further consideration, and the publication strategy. The initial face-to-face meetings were also used to initiate a large Delphi survey and to test the preceding versions of this TRIPOD-Cluster checklist.

A first draft of the TRIPOD-Cluster checklist was then prepared by the executive group. This draft formed the basis of an electronic questionnaire distributed to 77 Delphi participants, involving statisticians, epidemiologists, physicians, and journal editors with experience in prediction model research. The Delphi survey was opened in January 2019 and remained open for 50 days. A total of 27 modifications or new items to the original TRIPOD checklist were proposed to the Delphi participants, who rated each as “agree,” “no opinion,” or “disagree (please comment).” Three items had major disagreements (>30% scored “disagree”), and another three items had moderate disagreements (15-30% scored “disagree”), most of which were related to the (re)phrasing of newly proposed or existing items (appendix). Subsequently, a revised revision of the TRIPOD-Cluster checklist was prepared, and a second Delphi round among the same 77 participants was implemented in March 2019. All the attendant suggestions and results were then incorporated into the TRIPOD-Cluster checklist, which was discussed among participants of the annual conference of the International Society for Clinical Biostatistics (Leuven, 2019) to further refine the checklist.

How to use the TRIPOD-Cluster checklist and the main changes from the original TRIPOD checklist

The TRIPOD-Cluster checklist is not intended to be prescriptive about how prediction model studies using clustered data should be conducted or interpreted, although we believe that the detailed guidance for each item in the TRIPOD statement will help researchers and readers for this purpose. Although TRIPOD-Cluster is to be used as a standalone reporting guideline, we also recommend using TRIPOD-Cluster alongside the original TRIPOD explanation and elaboration document,12 and when applicable, the PROBAST (prediction model risk-of-bias assessment tool) guidance and its accompanying explanation and elaboration on the evaluation of risk bias in prediction models.4647 When prediction model studies do not account for clustering (eg, because cluster identifiers were not recorded or clusters were too small), we recommend that authors explicitly report this and adopt the original TRIPOD checklist.

Reporting of all relevant information might not always be feasible, for instance, owing to word count limits of specific journals or websites. In these situations, researchers might summarise the most relevant information in a table or figure, and provide additional details in the supplementary material.

A total of 19 original TRIPOD items and subitems were modified. Some subitems from the original TRIPOD checklist were merged in TRIPOD-Cluster for simplification. These merged subitems include the definition and assessment of outcomes (TRIPOD items 6a and 6b) and predictors (TRIPOD items 7a and 7b), characteristics of included data sources (TRIPOD items 4a, 4b, 5a, 5b, 13b, and 14a), and details on the model specification (TRIPOD items 15a and 15b).

Finally, 10 new subitems were included to look at identification of multiple eligible data sources or clusters (eg, in IPD-MA studies), data preparation, risk-of-bias assessment, heterogeneity in prediction model parameters, heterogeneity in prediction model performance estimates, and cluster specific and sensitivity analyses.

The resulting TRIPOD-Cluster guidance comprises 19 main items and is presented in this manuscript (table 3; separate version in the supplementary table). An explanation for each item and subitem is described in the accompanying explanation and elaboration document, where each subitem is illustrated with examples of good reporting from published electronic healthcare records (or multicentre) and IPD-MA studies, including those listed in table 1 and table 2.

Table 3

TRIPOD-Cluster checklist of items to include when reporting a study developing or validating a multivariable prediction model using clustered data

View this table:

Concluding remarks

TRIPOD-Cluster provides comprehensive consensus based guidance for the reporting of studies describing the development, validation, or updating of a multivariable diagnostic or prognostic prediction model using clustered data, such as those using electronic healthcare records or IPD-MA datasets (table 1 and table 2). TRIPOD-Cluster is not intended for reporting of studies in which the clustering is determined by repeated measurements of the same individuals. The checklist provides explicit guidance on the minimum required information to report, to help readers and potential model users understand how the study was designed, conducted, analysed, and inferred. We do not propose a standardised structure of reporting; rather, all the information requested in the checklist should be reported somewhere in the report or article, including supplementary material. We encourage authors to complete the checklist indicating the page number where the relevant information is reported and to submit this checklist with the article. The TRIPOD-Cluster checklist can be downloaded online (www.tripod-statement.org). Announcements relating to TRIPOD-Cluster will be made via www.tripod-statement.org and by the TRIPOD Twitter account (@TRIPODstatement). The EQUATOR Network will help disseminate and promote TRIPOD-Cluster.

Acknowledgments

We thank the members of the Delphi panel for their valuable input: Jeffrey Blume (Vanderbilt University, Nashville, TN, USA), Laura Bonnett (University of Liverpool, UK), Patrick Bossuyt (Amsterdam University Medical Centre, the Netherlands), Nancy Cook (Brigham and Women's Hospital, Boston, MA, USA), Carol Coupland (University of Nottingham, UK), Anneke Damen (University Medical Centre Utrecht, Utrecht University, the Netherlands), Joie Ensor (Keele University, UK), Frank Harrell (Vanderbilt University, Nashville, TN, USA), Georg Heinze (Medical University of Vienna, Austria), Martijn Heymans (Amsterdam University Medical Centre, the Netherlands), Lotty Hooft (Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, the Netherlands), Teus H Kappen (University Medical Centre Utrecht, Utrecht University, the Netherlands), Michael Kattan (Cleveland Clinic, Cleveland, OH, USA), Yannick Le Manach (McMaster University, Hamilton, ON, Canada), Glen Martin (University of Manchester, UK), Cynthia Mulrow (UT Health San Antonio Department of Medicine, San Antonio, TX, USA), Tri-Long Nguyen (University of Copenhagen, Denmark), Romin Pajouheshnia (Utrecht University, the Netherlands), Linda Peelen (University Medical Centre Utrecht, Utrecht University, the Netherlands), Willi Sauerbrei (University of Freiburg, Germany), Christopher Schmid (Brown University, Providence, RI, USA), Rob Scholten (Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, the Netherlands), Ewoud Schuit (University Medical Centre Utrecht, Utrecht University, the Netherlands), Ewout Steyerberg (Leiden University Medical Centre and Erasmus Medical Centre, the Netherlands), Yemisi Takwoingi (University of Birmingham, UK), Laure Wynants (Maastricht University, the Netherlands, and KU Leuven, Belgium).

We dedicate TRIPOD-Cluster to our friend Doug Altman, who co-initiated and was involved in TRIPOD-Cluster until his passing away on 3 June 2018.

Footnotes

  • Contributors: TPAD is the guarantor. TPAD, GSC, RDR, Doug Altman, JBR, and KGMM conceived the TRIPOD-Cluster statement and developed an initial checklist. TPAD organised two Delphi rounds to obtain feedback on individual items of the TRIPOD-Cluster checklist. TPAD and KGMM prepared the first draft of the manuscript. All authors reviewed subsequent drafts and assisted in revising the manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

  • Funding: No specific funding was given for this work. TPAD and JR acknowledge financial support from the Netherlands Organisation for Health Research and Development (grants 91617050 and 91215058). TPAD also acknowledges financial support from the European Union’s Horizon 2020 Research and Innovation Programme under ReCoDID Grant Agreement no. 825746. GSC is supported by the National Institute for Health and Care Research (NIHR) Biomedical Research Centre, Oxford, UK and Cancer Research UK (grant C49297/A27294). KIES is funded by the NIHR School for Primary Care Research (NIHR SPCR). BVC is supported by Research Foundation–Flanders (FWO; grants G0B4716N and G097322N) and Internal Funds KU Leuven (grants C24/15/037 and C24M/20/064). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR, the Department of Health and Social Care, or any other funding body. The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: GSC, KIES, BVC, JBR, and KGMM have no competing interests to disclose. RDR has received personal payments from universities and other organisations for providing training courses, and receives royalties for sales from two handbooks. TPAD provides consulting services to life science industries, including pharmaceutical companies and contract research organisations.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

http://creativecommons.org/licenses/by/4.0/

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.

References

  1. ↵
    1. Moons KGM,
    2. Royston P,
    3. Vergouwe Y,
    4. Grobbee DE,
    5. Altman DG
    . Prognosis and prognostic research: what, why, and how?BMJ2009;338:b375. doi:10.1136/bmj.b375 pmid:19237405
    OpenUrlFREE Full Text
  2. ↵
    1. Steyerberg EW,
    2. Moons KGM,
    3. van der Windt DA,
    4. et al.,
    5. PROGRESS Group
    . Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med2013;10:e1001381. doi:10.1371/journal.pmed.1001381 pmid:23393430
    OpenUrlCrossRefPubMed
  3. ↵
    1. Riley RD,
    2. van der Windt D,
    3. Croft P,
    4. Moons KGM
    , eds. Prognosis Research in Healthcare: concepts, methods, and impact.1st ed. Oxford University Press, 2019doi:10.1093/med/9780198796619.001.0001.
    OpenUrlCrossRef
  4. ↵
    1. Steyerberg EW
    . Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. 2nd ed. Springer International Publishing; 2019doi:10.1007/978-3-030-16399-0.
    OpenUrlCrossRefPubMed
  5. ↵
    1. Harrell FE
    . Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. 2nd ed. Springer; 2015doi:10.1007/978-3-319-19425-7.
    OpenUrlCrossRef
  6. ↵
    1. Bouwmeester W,
    2. Zuithoff NPA,
    3. Mallett S,
    4. et al
    . Reporting and methods in clinical prediction research: a systematic review. PLoS Med2012;9:1-12. doi:10.1371/journal.pmed.1001221 pmid:22629234
    OpenUrlCrossRefPubMed
  7. ↵
    1. Christodoulou E,
    2. Ma J,
    3. Collins GS,
    4. Steyerberg EW,
    5. Verbakel JY,
    6. Van Calster B
    . A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol2019;110:12-22. doi:10.1016/j.jclinepi.2019.02.004 pmid:30763612
    OpenUrlCrossRefPubMed
  8. ↵
    1. Altman DG,
    2. Vergouwe Y,
    3. Royston P,
    4. Moons KGM
    . Prognosis and prognostic research: validating a prognostic model. BMJ2009;338:b605. doi:10.1136/bmj.b605 pmid:19477892
    OpenUrlFREE Full Text
  9. ↵
    1. Royston P,
    2. Moons KGM,
    3. Altman DG,
    4. Vergouwe Y
    . Prognosis and prognostic research: Developing a prognostic model. BMJ2009;338:b604. doi:10.1136/bmj.b604 pmid:19336487
    OpenUrlFREE Full Text
  10. ↵
    1. Peat G,
    2. Riley RD,
    3. Croft P,
    4. et al.,
    5. PROGRESS Group
    . Improving the transparency of prognosis research: the role of reporting, data sharing, registration, and protocols. PLoS Med2014;11:e1001671. doi:10.1371/journal.pmed.1001671 pmid:25003600
    OpenUrlCrossRefPubMed
  11. ↵
    1. Collins GS,
    2. Reitsma JB,
    3. Altman DG,
    4. Moons KGM
    . Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med2015;162:55-63. doi:10.7326/M14-0697 pmid:25560714
    OpenUrlCrossRefPubMed
  12. ↵
    1. Moons KGM,
    2. Altman DG,
    3. Reitsma JB,
    4. et al
    . Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med2015;162:W1-73. doi:10.7326/M14-0698 pmid:25560730
    OpenUrlCrossRefPubMed
  13. ↵
    1. Debray TPA,
    2. Riley RD,
    3. Rovers MM,
    4. Reitsma JB,
    5. Moons KG,
    6. Cochrane IPD Meta-analysis Methods group
    . Individual participant data (IPD) meta-analyses of diagnostic and prognostic modeling studies: guidance on their use. PLoS Med2015;12:e1001886. doi:10.1371/journal.pmed.1001886 pmid:26461078
    OpenUrlCrossRefPubMed
  14. ↵
    1. Cook JA,
    2. Collins GS
    . The rise of big clinical databases. Br J Surg2015;102:e93-101. doi:10.1002/bjs.9723 pmid:25627139
    OpenUrlCrossRefPubMed
    1. Allotey J,
    2. Snell KIE,
    3. Chan C,
    4. et al.,
    5. IPPIC Collaborative Network
    . External validation, update and development of prediction models for pre-eclampsia using an Individual Participant Data (IPD) meta-analysis: the International Prediction of Pregnancy Complication Network (IPPIC pre-eclampsia) protocol. Diagn Progn Res2017;1:16. doi:10.1186/s41512-017-0016-z pmid:31093545
    OpenUrlCrossRefPubMed
    1. WHO CVD Risk Chart Working Group
    . World Health Organization cardiovascular disease risk charts: revised models to estimate risk in 21 global regions. Lancet Glob Health2019;7:e1332-45. doi:10.1016/S2214-109X(19)30318-3 pmid:31488387
    OpenUrlCrossRefPubMed
    1. Danesh J,
    2. Erqou S,
    3. Walker M,
    4. et al.,
    5. Emerging Risk Factors Collaboration
    . The Emerging Risk Factors Collaboration: analysis of individual data on lipid, inflammatory and other markers in over 1.1 million participants in 104 prospective studies of cardiovascular diseases. Eur J Epidemiol2007;22:839-69. doi:10.1007/s10654-007-9165-7 pmid:17876711
    OpenUrlCrossRefPubMedWeb of Science
    1. Wilder-Smith A,
    2. Wei Y,
    3. Araújo TVB,
    4. et al.,
    5. Zika Virus Individual Participant Data Consortium
    . Understanding the relation between Zika virus infection during pregnancy and adverse fetal, infant and child outcomes: a protocol for a systematic review and individual participant data meta-analysis of longitudinal studies of pregnant women and their infants and children. BMJ Open2019;9:e026092. doi:10.1136/bmjopen-2018-026092 pmid:31217315
    OpenUrlAbstract/FREE Full Text
    1. Geersing GJ,
    2. Zuithoff NPA,
    3. Kearon C,
    4. et al
    . Exclusion of deep vein thrombosis using the Wells rule in clinically important subgroups: individual patient data meta-analysis. BMJ2014;348:g1340. doi:10.1136/bmj.g1340 pmid:24615063
    OpenUrlAbstract/FREE Full Text
    1. Steyerberg EW,
    2. Mushkudiani N,
    3. Perel P,
    4. et al
    . Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med2008;5:e165, discussion e165. doi:10.1371/journal.pmed.0050165 pmid:18684008
    OpenUrlCrossRefPubMed
    1. Marmarou A,
    2. Lu J,
    3. Butcher I,
    4. et al
    . IMPACT database of traumatic brain injury: design and description. J Neurotrauma2007;24:239-50. doi:10.1089/neu.2006.0036 pmid:17375988
    OpenUrlCrossRefPubMedWeb of Science
    1. Hippisley-Cox J,
    2. Stables D,
    3. Pringle M
    . QRESEARCH: a new general practice database for research. Inform Prim Care2004;12:49-50.pmid:15140353
    OpenUrlPubMed
    1. Hippisley-Cox J,
    2. Coupland C,
    3. Brindle P
    . Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: prospective cohort study. BMJ2017;357:j2099. doi:10.1136/bmj.j2099 pmid:28536104
    OpenUrlAbstract/FREE Full Text
    1. Hammill SC,
    2. Stevenson LW,
    3. Kadish AH,
    4. et al
    . Review of the registry’s first year, data collected, and future plans. Heart Rhythm2007;4:1260-3. doi:10.1016/j.hrthm.2007.07.021 pmid:17765637
    OpenUrlCrossRefPubMedWeb of Science
    1. Dodson JA,
    2. Reynolds MR,
    3. Bao H,
    4. et al.,
    5. NCDR
    . Developing a risk model for in-hospital adverse events following implantable cardioverter-defibrillator implantation: a report from the NCDR (National Cardiovascular Data Registry). J Am Coll Cardiol2014;63:788-96. doi:10.1016/j.jacc.2013.09.079 pmid:24333491
    OpenUrlFREE Full Text
    1. Denaxas SC,
    2. George J,
    3. Herrett E,
    4. et al
    . Data resource profile: cardiovascular disease research using linked bespoke studies and electronic health records (CALIBER). Int J Epidemiol2012;41:1625-38. doi:10.1093/ije/dys188 pmid:23220717
    OpenUrlCrossRefPubMedWeb of Science
    1. Rapsomaniki E,
    2. Shah A,
    3. Perel P,
    4. et al
    . Prognostic models for stable coronary artery disease based on electronic health record cohort of 102 023 patients. Eur Heart J2014;35:844-52. doi:10.1093/eurheartj/eht533 pmid:24353280
    OpenUrlCrossRefPubMedWeb of Science
    1. Grant SW,
    2. Hickey GL,
    3. Grayson AD,
    4. Mitchell DC,
    5. McCollum CN
    . National risk prediction model for elective abdominal aortic aneurysm repair. Br J Surg2013;100:645-53. doi:10.1002/bjs.9047 pmid:23338659
    OpenUrlCrossRefPubMed
  29. ↵
    1. Localio AR,
    2. Berlin JA,
    3. Ten Have TR,
    4. Kimmel SE
    . Adjustments for center in multicenter studies: an overview. Ann Intern Med2001;135:112-23. doi:10.7326/0003-4819-135-2-200107170-00012 pmid:11453711
    OpenUrlCrossRefPubMedWeb of Science
  30. ↵
    1. Riley RD,
    2. Tierney JF,
    3. Stewart LA
    , eds. Individual participant data meta-analysis: a handbook for healthcare research. Wiley; 2021doi:10.1002/9781119333784.
    OpenUrlCrossRef
  31. ↵
    1. Debray TPA,
    2. de Jong VMT,
    3. Moons KGM,
    4. Riley RD
    . Evidence synthesis in prognosis research. Diagn Progn Res2019;3:13. doi:10.1186/s41512-019-0059-4 pmid:31338426
    OpenUrlCrossRefPubMed
  32. ↵
    1. Campbell MK,
    2. Elbourne DR,
    3. Altman DG,
    4. CONSORT group
    . CONSORT statement: extension to cluster randomised trials. BMJ2004;328:702-8. doi:10.1136/bmj.328.7441.702 pmid:15031246
    OpenUrlFREE Full Text
  33. ↵
    1. Collins GS,
    2. Ogundimu EO,
    3. Altman DG
    . Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med2016;35:214-26. doi:10.1002/sim.6787 pmid:26553135
    OpenUrlCrossRefPubMed
  34. ↵
    1. Allotey J,
    2. Snell KI,
    3. Smuk M,
    4. et al
    . Validation and development of models using clinical, biochemical and ultrasound markers for predicting pre-eclampsia: an individual participant data meta-analysis. Health Technol Assess2020;24:1-252. doi:10.3310/hta24720 pmid:33336645
    OpenUrlCrossRefPubMed
  35. ↵
    1. Meads C,
    2. Ahmed I,
    3. Riley RD
    . A systematic review of breast cancer incidence risk prediction models with meta-analysis of their performance. Breast Cancer Res Treat2012;132:365-77. doi:10.1007/s10549-011-1818-2 pmid:22037780
    OpenUrlCrossRefPubMed
  36. ↵
    1. Damen JA,
    2. Pajouheshnia R,
    3. Heus P,
    4. et al
    . Performance of the Framingham risk models and pooled cohort equations for predicting 10-year risk of cardiovascular disease: a systematic review and meta-analysis. BMC Med2019;17:109. doi:10.1186/s12916-019-1340-7 pmid:31189462
    OpenUrlCrossRefPubMed
  37. ↵
    1. Debray TPA,
    2. Vergouwe Y,
    3. Koffijberg H,
    4. Nieboer D,
    5. Steyerberg EW,
    6. Moons KGM
    . A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol2015;68:279-89. doi:10.1016/j.jclinepi.2014.06.018 pmid:25179855
    OpenUrlCrossRefPubMed
  38. ↵
    1. Van Calster B,
    2. Nieboer D,
    3. Vergouwe Y,
    4. De Cock B,
    5. Pencina MJ,
    6. Steyerberg EW
    . A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol2016;74:167-76. doi:10.1016/j.jclinepi.2015.12.005 pmid:26772608
    OpenUrlCrossRefPubMed
  39. ↵
    1. Debray TPA,
    2. Moons KGM,
    3. Ahmed I,
    4. Koffijberg H,
    5. Riley RD
    . A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis. Stat Med2013;32:3158-80. doi:10.1002/sim.5732 pmid:23307585
    OpenUrlCrossRefPubMed
  40. ↵
    1. Wynants L,
    2. Vergouwe Y,
    3. Van Huffel S,
    4. Timmerman D,
    5. Van Calster B
    . Does ignoring clustering in multicenter data influence the performance of prediction models? A simulation study. Stat Methods Med Res2018;27:1723-36. doi:10.1177/0962280216668555 pmid:27647815
    OpenUrlCrossRefPubMed
  41. ↵
    1. de Jong VMT,
    2. Moons KGM,
    3. Eijkemans MJC,
    4. Riley RD,
    5. Debray TPA
    . Developing more generalizable prediction models from pooled studies and large clustered data sets. Stat Med2021;40:3533-59. doi:10.1002/sim.8981 pmid:33948970
    OpenUrlCrossRefPubMed
  42. ↵
    1. Moher D,
    2. Schulz KF,
    3. Simera I,
    4. Altman DG
    . Guidance for developers of health research reporting guidelines. PLoS Med2010;7:e1000217. doi:10.1371/journal.pmed.1000217 pmid:20169112
    OpenUrlCrossRefPubMed
  43. ↵
    1. Stewart LA,
    2. Clarke M,
    3. Rovers M,
    4. et al.,
    5. PRISMA-IPD Development Group
    . Preferred Reporting Items for Systematic Review and Meta-Analyses of individual participant data: the PRISMA-IPD Statement. JAMA2015;313:1657-65. doi:10.1001/jama.2015.3656 pmid:25919529
    OpenUrlCrossRefPubMed
  44. ↵
    1. Vandenbroucke JP,
    2. von Elm E,
    3. Altman DG,
    4. et al.,
    5. STROBE Initiative
    . Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. PLoS Med2007;4:e297. doi:10.1371/journal.pmed.0040297 pmid:17941715
    OpenUrlCrossRefPubMed
  45. ↵
    1. Benchimol EI,
    2. Smeeth L,
    3. Guttmann A,
    4. et al.,
    5. RECORD Working Committee
    . The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement. PLoS Med2015;12:e1001885. doi:10.1371/journal.pmed.1001885 pmid:26440803
    OpenUrlCrossRefPubMed
  46. ↵
    1. Moons KGM,
    2. Wolff RF,
    3. Riley RD,
    4. et al
    . PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration. Ann Intern Med2019;170:W1-33. doi:10.7326/M18-1377 pmid:30596876
    OpenUrlCrossRefPubMed
  47. ↵
    1. Wolff RF,
    2. Moons KGM,
    3. Riley RD,
    4. et al.,
    5. PROBAST Group†
    . PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Ann Intern Med2019;170:51-8. doi:10.7326/M18-1376 pmid:30596875
    OpenUrlCrossRefPubMed
Back to top