Papers

Commentary: Prognostic models: clinically useful or quickly forgotten?

^a Medical Informatics, Biomedical Informatics Unit, Imperial Cancer Research Fund, PO Box 123, London WC2A 3PX, ^b ICRF Medical Statistics Group, Centre for Statistics in Medicine, Institute for Health Sciences, Oxford OX3 7LF

We are all familiar with using single items of patient data such as age or smoking history to help in making difficult clinical decisions.¹ Prognostic models are more complex tools for helping decision making that combine two or more items of patient data to predict clinical outcomes.² They are of potential value when doctors are making difficult clinical decisions (such as ordering invasive tests or selecting which patients should benefit from scarce resources),³ conducting comparative audit,⁴ or selecting uniform groups of patients for clinical trials.⁵ Another prognostic model appears in this week's BMJ⁶ to join the hundreds published every year.⁷ However, apart from exceptions such as the Glasgow coma scale⁸ and APACHE III,⁹ few of these models are routinely used to inform difficult clinical decisions.

It might be argued that doctors never prognosticate, working always in the present, but studies of medical decision making show this is untrue.^{10 11} Some doctors might claim that they can foretell patient outcomes better than any statistical model, but again there is contrary evidence.^{9 12} A journal editor's view might be that most published models reflect preliminary work and need further research before clinical adoption.¹³ Finally, some models predict events that are of no clinical relevance or do not generate predictions in time to inform clinical decisions, suggesting that their developers wished merely to publish journal articles, not build clinically useful tools.¹⁴

We believe that the main reasons why doctors reject published prognostic models are lack of clinical credibility and lack of evidence that a prognostic model can support decisions about patient care (that is, evidence of accuracy, generality, and effectiveness). We examine each of these issues in turn.

Clinical credibility of the model

However accurate a model is in statistical terms, doctors will be reluctant to use it to inform their patient management decisions unless they believe in the model and its predictions. Some prerequisites for clinical credibility include:

* All clinically relevant patient data should have been tested for inclusion in the model. For example, it would be foolish to omit smoking history from the variables screened when constructing a model to predict the probability of acute myocardial infarction

* It should be simple for doctors to obtain all the patient data required, reliably and without expending undue resources, in time to generate the prediction and guide decisions. Data should be obtainable with high reliability, particularly in those patients for which the model's prediction are most likely to be needed. This is not always the case¹⁵

* Model builders should try to avoid arbitrary thresholds for continuous variables. For example, it is unlikely that the prognosis for a woman with an ulcerated melanoma 3.9 mm thick would be very different if it was 4 mm thick. (Aitchison et al⁶ discuss this issue in their paper.⁶)

* The model's structure should be apparent and its predictions should make sense to the doctors who will rely on them, as only then will the law treat users of the model as "learned intermediaries" in a case of alleged negligence.¹⁶ This requirement means that the statistical modelling method must be correctly applied and not transgress the method's assumptions; these were checked in only a fifth of one series of published models.¹⁷ It also makes "black box" models such as artificial neural networks less suitable for clinical applications¹⁸

* It should be simple for doctors to calculate the model's prediction for a patient. Thus, a model which takes the form of a printed tree⁶ or clinical algorithm¹⁹ will probably be used more often than one requiring data entry to a computer to make complex calculations.²⁰

Evidence of accuracy

A prognostic model is unlikely to be useful unless its predictions are at least as accurate as those of the doctors who would use it. A low error rate alone, however, is not enough, as there are two kinds of errors with different clinical consequences. Thus, the model should rarely fail to predict an event that will occur (have a low false negative rate) and also seldom mistakenly predict it when it will not occur (have a low false positive rate). These error rates should be checked on a large test set of cases in which occurrence or absence of the event has already been reliably determined²¹ and which has not been used to derive the model.²² "Large" means that there should be at least five²² to ten²³ test cases in which the outcome being predicted (such as death) occurs per item of clinical data used to predict the outcome.

If the model predicts an outcome with a probability (for example, "40% mortality") the probability should also be accurate, meaning that 40% of the patients in whom such a prediction is issued do in fact die.²⁴ Few models are shown to be both accurate and "well calibrated" in this way (such as Franklin et al,¹⁹ Murray et al,²⁵ MRC Antiepileptic Drug Withdrawal Study Group²⁶), but both kinds of accuracy are necessary if doctors are to rely on predictions when making difficult decisions.

Evidence of generality

Some doctors believe that no prognostic model derived from one population can be generalised to patients drawn from another,²⁷ in the same way that some deny that clinical trials or overviews can inform individual decisions about treatment. We and others^{21 22} believe that model predictions cannot be safely applied to other patients unless

* There has been separate testing--in time and place--of the model on a new test set. In view of the established difficulty of transferring prognostic models,²⁸ it is surprising that such follow up testing is seldom performed. Use of statistical techniques such as shrinkage of the regression coefficients²⁹ to correct for overoptimism in the model (for example, MRC Antiepileptic Drug Withdrawal Study Group²⁶) may help to make models more transferable

* Each item of data used as input to the model has been defined clearly using definitions that reflect widespread use.¹ Even apparently obvious items like age need clarification: does this mean the patient's age at the start of symptoms, at diagnosis, or when the prognosis is required?

* The model has been derived and validated using a defined "inception cohort" of patients²¹ whose selection criteria describe the patients to which doctors may safely apply the model. This implies that both development and testing of models should take place prospectively according to a protocol,³⁰ not retrospectively using existing databases with all their biases.³¹

Evidence of clinical effectiveness

No one would prescribe routine drugs on the basis of in vitro testing alone: clinical trials are vital to evaluate safety and efficacy. Even in the case of an accurate credible prognostic model, doctors should demand empirical evidence from well conducted clinical trials that the model is clinically effective. These trials should be controlled studies in which the effects of providing predictions on actual clinical practices or patient outcomes are measured. The studies must be carefully designed to eliminate biases such as the checklist effect, Hawthorne effect, and contamination.³² Although such studies require considerable resources, they are as necessary for prognostic models as are phase 3 randomised trials for drugs.^{33 34} There have been few trials of prognostic models (for example, Murray et al³⁵ and Goldman et al³⁶). There have, however, been at least 28 trials of computerised decision support systems, a similar technology, and an overview of these trials showed that many had a beneficial impact on clinical practice or patient outcome.²

Conclusion

s

Since only a handful of prognostic models have yet accrued adequate evidence of accuracy, generality, and effectiveness, doctors' poor uptake of these tools to aid difficult decisions may result from healthy scepticism of undocumented new technology. A recent development that may help to improve clinical acceptability is combining the prognosis generated by the model with the doctor's own estimate of prognosis, since this may add to the model's performance.³⁷ Another is modification of the model in the light of clinical experience.³⁸

Given the increasing complexity of the methods used to develop prognostic models⁷ and their potential power to influence clinical decisions,² it is reassuring that the model described in this week's BMJ was developed and evaluated by statisticians working in close collaboration with doctors.⁶ This pattern of collaboration and adherence to the principles described in this paper seem likely to yield models that will be useful to doctors making difficult decisions.

JCW acknowledges the many useful insights resulting from collaboration with David Spiegelhalter of the MRC Biostatistics Unit, Cambridge.

Funding: None.

Conflict of interest: None.

1. Wyatt JC. Clinical data systems. Part 1: data and medical records. Lancet 1994;344:1543-7. [Medline]
2. Johnston ME, Langton KB, Haynes RB, Matthieu D. A critical appraisal of research on the effects of computer-based decision support systems on clinician performance and patient outcomes. Ann Intern Med 1994;120:135-42. [Abstract/Free Full Text]
3. Holmes L, Loughead K, Treasure T, Gallivan S. Which patients will not benefit from further intensive care after cardiac surgery? Lancet 1994;344:1200-2.
4. Rowan KM, Kerr JH, McPherson K, Short A, Vessey MP. Intensive Care Society's APACHE II study in Britain and Ireland--II: Outcome comparisons of intensive care units after adjustment for case mix by the American APACHE II method. BMJ 1993;307:977-81.
5. Riethmuller G, Schneider-Gadicke W, Schlimok G, Schmiegel W, Raab R, Hoffken K, et al. Randomised trial of, monoclonal antibody for adjuvant therapy of resected Dukes' C colorectal carcinoma. Lancet 1994;343:1177-83. [Medline]
6. Aitchison TC, Sirel JM, Watt DC, MacKie RM. Prognostic trees to aid prognosis in patients with cutaneous malignant melanoma. BMJ 1995;311:1536-9. [Abstract/Free Full Text]
7. Concato J, Feinstein AR, Holford TR. The risk of determining risk with multivariable models. Ann Intern Med 1993;118:201-10. [Abstract/Free Full Text]
8. Teasdale C, Jennett B. Assessment of coma and impaired consciousness: a practical scale. Lancet 1974;ii:81-4.
9. Knaus WA, Wagner DP, Lynn J. Short term mortality predictions for critically ill hospitalised adults: science and ethics. Science 1991;254:389-94. [Abstract/Free Full Text]
10. Elstein AS, Shulman LS, Sprafka SA. Medical problem solving. Cambridge, MA: Harvard University Press, 1978.
11. Patel V, Evans D, Groen G. Biomedical knowledge and clinical reasoning. In: Evans D, Patel V, eds. Cognitive science in medicine. Cambridge, MA: MIT Press, 1989:53-112.
12. Lee KL, Pryor DB, Harrell FE, Califf HM, Behar VS, Floyd PL, et al. Predicting outcome in coronary disease: statistical models vs. expert clinicians. Am J Med 1986;80:553-60. [Medline]
13. Haynes RB. Loose connections between peer-reviewed clinical journals and clinical practice. Ann Intern Med 1990;113:724-8.
14. Altman DG. The scandal of poor medical research. BMJ 1994;308:283-4. [Free Full Text]
15. Rowley G, Fielding K. Reliability and accuracy of the Glasgow coma scale with experienced and inexperienced users. Lancet 1991;337:535-8. [Medline]
16. Brahams D, Wyatt J. Decision-aids and the law. Lancet 1989;ii:632-4.
17. Coste J, Fermanian J, Venot A. Methodological and statistical problems in the construction of composite measurement scales: a survey of six medical and epidemiological journals. Stat Med 1995;14:331-45. [Medline]
18. Hart A, Wyatt J. Evaluating black boxes as medical decision-aids: issues arising from a study of neural networks. Med Inf (Lond) 1990;15:229-36.
19. Franklin RCG, Spiegelhalter DJ, Macartney F, Bull K. Evaluation of a diagnostic algorithm for heart disease in neonates. BMJ 1991;302:935-9.
20. Heathfield HA, Wyatt IC. Philosophies for the design and development of clinical decision-support systems. Methods Inf Med 1993;32:1-8. [Medline]
21. Sackett D, Haynes R, Guyatt G, Tugwell P. Clinical epidemiology: a basic science for clinical medicine. 2nd ed. Boston, MA: Little Brown, 1991:173-85.
22. Wasson JH, Sox HC, Neff RK, Goldman L. Clinical prediction rules: applications and methodological standards. N Engl J Med 1985;313:793-9. [Abstract]
23. Harrell FE, Lee KL, Matchar DB, Reichert TA. Regression models for prognostic, prediction: advantages, problems and suggested solutions. Cancer Treatment Reports 1985;69:1071-7. [Medline]
24. Wyatt J, Spiegelhalter D. Evaluating medical expert systems: what to test and how? Med Inf (Lond) 1990;15:205-17.
25. Murray. GD, Murray LS, Barlow P, Teasdale GM, Jennett WB. Assessing the performance and clinical impact of a computerised prognostic system in severe head injury. Stat Med 1986;5:403-10. [Medline]
26. Medical Research Council Antiepileptic Drug Withdrawal Study Group. Prognostic index for recurrence of seizures after remission of epilepsy. BMJ 1993;306:1374-8.
27. Pilkington SN. APACHE scoring and prediction of survival in intensive care. BMJ 1995;310:1197. [Free Full Text]
28. Centor AM, Yarbrough B, Wood JP. Inability to predict relapse in acute asthma. N Engl J Med 1984;310:577-80. [Medline]
29. Phillips AN, Thompson SG, Pocock SJ. Prognostic scores for detecting a high risk group: estimating the sensitivity when applied to new data. Stat Med 1990;9:1189-98. [Medline]
30. Arnbjornsson E. Scoring system for computer-aided diagnosis of acute appendicitis: the value of prospective versus retrospective studies. Ann Chir Gynaecol 1985;74:159-66. [Medline]
31. Wyatt JC. Acquisition and use of clinical data for audit and research. Journal of Evaluation in Clinical Practice 1995;1:15-27. [Medline]
32. Wyatt J, Spiegelhalter D. Field trials of medical decision-aids: potential problems and solutions. In: Clayton P, ed. Proceedings of the 15th symposium on computer applications in medical care, Washington 1991. New York: McGraw Hill, 1991:3-7.
33. Spiegelhalter DJ. Evaluation of medical decision-aids, with an application to a system for dyspepsia. Stat Med 1983;2;207-16.
34. Simon R, Altman DG. Statistical aspects of prognostic factor studies in oncology, Br J Cancer 1994;69:979-85.
35. Murray LS, Teasciale GM, Murray GD, Jennett B, Miller JD, Pickard JD, et al. Does prediction of outcome alter patient management? Lancet 1993;341:1487-91.
36. Goldman L, Cook EF, Brand DA, Lee TH, Rouan GW, Weisberg MC, et al. A computer protocol to predict myocardial infarction in emergency room patients. N Engl J Med 1988;318:797-803. [Abstract]
37. Knaus WA, Harrell FE, Lynne J, Goldman L, Phillips RS, Connors AF, et al. The SUPPORT prognostic model: objective estimates of survival for seriously ill, hospitalised patients. Ann Intern Med 1995;122:191-203. [Abstract/Free Full Text]
38. Van Houwelingen H, Thorogood J. Construction, validation and updating of a prognostic model for kidney graft survival. Stat Med 1995;14:1999-2008. [Medline]