Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisalBMJ 2020; 369 doi: https://doi.org/10.1136/bmj.m1328 (Published 07 April 2020) Cite this as: BMJ 2020;369:m1328
- Laure Wynants, assistant professor1 2,
- Ben Van Calster, associate professor2 3,
- Gary S Collins, professor4 5,
- Richard D Riley, professor6,
- Georg Heinze, associate professor7,
- Ewoud Schuit, assistant professor8 9,
- Marc M J Bonten, professor8 10,
- Johanna A A Damen , assistant professor8 9,
- Thomas P A Debray, assistant professor8 9,
- Maarten De Vos, associate professor2 11,
- Paula Dhiman, research fellow4 5,
- Maria C Haller, medical doctor7 12,
- Michael O Harhay, assistant professor13 14,
- Liesbet Henckaerts, assistant professor15 16,
- Nina Kreuzberger, research associate17,
- Anna Lohmann, researcher in training18,
- Kim Luijken, doctoral candidate18,
- Jie Ma, medical statistician5,
- Constanza L Andaur Navarro, doctoral student8 9,
- Johannes B Reitsma, associate professor8 9,
- Jamie C Sergeant, senior lecturer19 20,
- Chunhu Shi, research associate21,
- Nicole Skoetz, medical doctor17,
- Luc J M Smits, professor1,
- Kym I E Snell, lecturer6,
- Matthew Sperrin, senior lecturer22,
- René Spijker, information specialist8 9,
- Ewout W Steyerberg, professor3,
- Toshihiko Takada, assistant professor4,
- Sander M J van Kuijk, research fellow23,
- Florien S van Royen, research fellow8,
- Christine Wallisch, research fellow7 24 25,
- Lotty Hooft, associate professor8 9,
- Karel G M Moons, professor8 9,
- Maarten van Smeden, assistant professor8
- 1Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Peter Debyeplein 1, 6229 HA Maastricht, Netherlands
- 2Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- 3Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, Netherlands
- 4Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Musculoskeletal Sciences, University of Oxford, Oxford, UK
- 5NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford, UK
- 6Centre for Prognosis Research, School of Primary, Community and Social Care, Keele University, Keele, UK
- 7Section for Clinical Biometrics, Centre for Medical Statistics, Informatics and Intelligent Systems, Medical University of Vienna, Vienna, Austria
- 8Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
- 9Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
- 10Department of Medical Microbiology, University Medical Centre Utrecht, Utrecht, Netherlands
- 11Department of Electrical Engineering, ESAT Stadius, KU Leuven, Leuven, Belgium
- 12Ordensklinikum Linz, Hospital Elisabethinen, Department of Nephrology, Linz, Austria
- 13Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- 14Palliative and Advanced Illness Research (PAIR) Center and Division of Pulmonary and Critical Care Medicine, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- 15Department of Microbiology, Immunology and Transplantation, KU Leuven-University of Leuven, Leuven, Belgium
- 16Department of General Internal Medicine, KU Leuven-University Hospitals Leuven, Leuven, Belgium
- 17Evidence-Based Oncology, Department I of Internal Medicine and Center for Integrated Oncology Aachen Bonn Cologne Dusseldorf, Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany
- 18Department of Clinical Epidemiology, Leiden University Medical Centre, Leiden, Netherlands
- 19Centre for Biostatistics, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
- 20Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
- 21Division of Nursing, Midwifery and Social Work, School of Health Sciences, University of Manchester
- 22Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK
- 23Department of Clinical Epidemiology and Medical Technology Assessment, Maastricht University Medical Centre+, Maastricht, Netherlands
- 24Charité Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, Berlin, Germany
- 25Berlin Institute of Health, Berlin, Germany
- Correspondence to: L Wynants
- Accepted 31 March 2020
- Final version accepted 4 May 2020
Objective To review and critically appraise published and preprint reports of prediction models for diagnosing coronavirus disease 2019 (covid-19) in patients with suspected infection, for prognosis of patients with covid-19, and for detecting people in the general population at increased risk of becoming infected with covid-19 or being admitted to hospital with the disease.
Design Living systematic review and critical appraisal.
Data sources PubMed and Embase through Ovid, Arxiv, medRxiv, and bioRxiv up to 7 April 2020.
Study selection Studies that developed or validated a multivariable covid-19 related prediction model.
Data extraction At least two authors independently extracted data using the CHARMS (critical appraisal and data extraction for systematic reviews of prediction modelling studies) checklist; risk of bias was assessed using PROBAST (prediction model risk of bias assessment tool).
Results 4909 titles were screened, and 51 studies describing 66 prediction models were included. The review identified three models for predicting hospital admission from pneumonia and other events (as proxy outcomes for covid-19 pneumonia) in the general population; 47 diagnostic models for detecting covid-19 (34 were based on medical imaging); and 16 prognostic models for predicting mortality risk, progression to severe disease, or length of hospital stay. The most frequently reported predictors of presence of covid-19 included age, body temperature, signs and symptoms, sex, blood pressure, and creatinine. The most frequently reported predictors of severe prognosis in patients with covid-19 included age and features derived from computed tomography scans. C index estimates ranged from 0.73 to 0.81 in prediction models for the general population, from 0.65 to more than 0.99 in diagnostic models, and from 0.85 to 0.99 in prognostic models. All models were rated at high or unclear risk of bias, mostly because of non-representative selection of control patients, exclusion of patients who had not experienced the event of interest by the end of the study, high risk of model overfitting, and vague reporting. Most reports did not include any description of the study population or intended use of the models, and calibration of the model predictions was rarely assessed.
Conclusion Prediction models for covid-19 are quickly entering the academic literature to support medical decision making at a time when they are urgently needed. This review indicates that proposed models are poorly reported, at high risk of bias, and their reported performance is probably optimistic. Hence, we do not recommend any of these reported prediction models to be used in current practice. Immediate sharing of well documented individual participant data from covid-19 studies and collaboration are urgently needed to develop more rigorous prediction models, and validate promising ones. The predictors identified in included models should be considered as candidate predictors for new models. Methodological guidance should be followed because unreliable predictions could cause more harm than benefit in guiding clinical decisions. Finally, studies should adhere to the TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) reporting guideline.
Readers’ note This article is a living systematic review that will be updated to reflect emerging evidence. Updates may occur for up to two years from the date of original publication. This version is update 1 of the original article published on 7 April 2020 (BMJ 2020;369:m1328), and previous updates can be found as data supplements (https://www.bmj.com/content/369/bmj.m1328/related#datasupp).
Contributors: LW conceived the study. LW and MvS designed the study. LW, MvS, and BVC screened titles and abstracts for inclusion. LW, BVC, GSC, TPAD, MCH, GH, KGMM, RDR, ES, LJMS, EWS, KIES, CW, JAAD, PD, MCH, NK, AL, KL, JM, CLAN, JBR, JCS, CS, NS, MS, RS, TT, SMJvK, FSvR, LH, and MvS extracted and analysed data. MDV helped interpret the findings on deep learning studies and MMJB, LH, and MCH assisted in the interpretation from a clinical viewpoint. RS and FSvR offered technical and administrative support. LW and MvS wrote the first draft, which all authors revised for critical content. All authors approved the final manuscript. LW and MvS are the guarantors. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
Funding: LW is a postdoctoral fellow of Research Foundation–Flanders (FWO). BVC received support from FWO (grant G0B4716N) and Internal Funds KU Leuven (grant C24/15/037). TPAD acknowledges financial support from the Netherlands Organisation for Health Research and Development (grant No 91617050). KGMM and JAAD gratefully acknowledge financial support from Cochrane Collaboration (SMF 2018). KIES is funded by the National Institute for Health Research School for Primary Care Research (NIHR SPCR). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care. GSC was supported by the NIHR Biomedical Research Centre, Oxford, and Cancer Research UK (programme grant C49297/A27294). JM was supported by the Cancer Research UK (programme grant C49297/A27294). PD was supported by the NIHR Biomedical Research Centre, Oxford. MOH is supported by the National Heart, Lung, and Blood Institute of the United States National Institutes of Health (grant No R00 HL141678). The funders played no role in study design, data collection, data analysis, data interpretation, or reporting. The guarantors had full access to all the data in the study, take responsibility for the integrity of the data and the accuracy of the data analysis, and had final responsibility for the decision to submit for publication.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; no competing interests with regards to the submitted work; LW discloses support from Research Foundation–Flanders (FWO); RDR reports personal fees as a statistics editor for The BMJ (since 2009), consultancy fees for Roche for giving meta-analysis teaching and advice in October 2018, and personal fees for delivering in-house training courses at Barts and The London School of Medicine and Dentistry, and also the Universities of Aberdeen, Exeter, and Leeds, all outside the submitted work; MS coauthored the editorial on the original article.
Ethical approval: Not required.
Data sharing: The study protocol is available online at https://osf.io/ehc47/. Most included studies are publicly available. Additional data are available upon reasonable request.
The lead authors affirm that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.
Dissemination to participants and related patient and public communities: The study protocol is available online at https://osf.io/ehc47/.
This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.