Prognostic modelling in traumatic brain injury

BMJ 2008; 336 doi: http://dx.doi.org/10.1136/bmj.39461.616991.80 (Published 21 February 2008) Cite this as: BMJ 2008;336:397

Can reliably estimate the probability of outcomes for groups but not for individuals

Hippocrates is said to have remarked in 400 BC that “No head injury is too severe to despair of, nor too trivial to ignore.” While this prognostic scheme achieves absolute accuracy, its precision leaves something to be desired. More recently, many groups have attempted to produce more detailed risk adjustment models for predicting outcome in traumatic brain injury. In 2006, a systematic review concluded that most predictive models were inadequately validated, poorly presented, and based on studies from single centres with small samples that excluded patients from low income countries (where traumatic brain injury is most common).1 In the accompanying paper, the Medical Research Council CRASH Trial Collaborators provide a series of prognostic models that attempt to remedy these shortcomings.2

Their models were developed on clinical data from the 10 008 patients recruited to a trial of corticosteroids in traumatic brain injury.3 4 Separate variants of the models allow the option of including imaging data from computed tomography, and of selecting data on predicting outcomes for high income and low to middle income countries. The models have been made publicly available on a web based calculator, which allows entry of clinical and imaging data to produce an estimated risk of death or disability.

The CRASH models perform well within the population used for their development. They show high discrimination of the overall probability of a poor outcome (C statistic >0.8) and good calibration (measured by the degree of concordance between a range of predicted and observed probabilities of poor outcome). However, the generalisability of any model depends on validating its accuracy in a separate test population. The authors have conducted such an external validation in the IMPACT database,5 which includes 8509 patients with moderate and severe traumatic brain injury in randomised controlled trials and observational studies conducted between 1984 and 1997. While the models continued to show reasonable discrimination in this exercise (C statistic 0.77), calibration was less well preserved for some of the models. Despite this, the CRASH models are an improvement on earlier attempts at predicting outcome in traumatic brain injury, and the authors suggest that the models may help in clinical trial design, comparative audit, and clinical decision making.

Use of the CRASH models is likely to improve the design of trials, given the poor record of neuroprotective trials in traumatic brain injury.6 Many patients included in such trials have expected outcomes that are so irreversibly favourable or unfavourable that no intervention could realistically be expected to have an effect. Better stratification of patients at entry could greatly improve the balance between the treatment arms, particularly in small trials, and adjustment for baseline risk in trial analyses can improve precision and statistical power. An additional advantage of robust outcome prediction in this context would be the use of a technique called sliding dichotomy to assess the benefit of interventions.7 Unlike conventional dichotomisation of outcomes, which compares the overall number of favourable outcomes between treatment and control arms, this approach compares predicted and observed outcomes on a case by case basis in two arms of a trial to detect significant treatment effects. Such an approach increases the efficiency of a trial and may greatly reduce the required sample size.8

These arguments also apply to comparative clinical audit, but with some caveats. Important limitations arise from the CRASH models being based on data from a randomised clinical trial (albeit a pragmatic trial). Well known limitations of randomised controlled trials include the effect of using inclusion criteria, the logistics of recruiting participants with sufficiently severe disease, and the possibility that patient outcomes tend to be better in randomised controlled trials, even in control groups.9 Consequently, the outcome standards provided by such prognostic models should be validated in a “real world” setting and compared against existing schemes before they are used for comparative benchmarking.10 Risk adjustment models may retain discrimination when transferred between clinical contexts, but their calibration—the degree of concordance between predicted and observed probabilities of poor outcome—often deteriorates (as was the case in the CRASH models). It is best to develop such models in the same context in which they will be used for clinical audit. Despite these reservations, in the context of trial design and clinical audit, the outcome predictions provided by prognostic schemes are applied to groups rather than individuals, and hence are relatively safe.

Far greater caution is needed if such a model is to be used for making decisions about treatment in individual patients. Estimates of outcome probability from the 10 000 patients in the CRASH trial are based on collective clinical experience beyond that achievable by any individual clinician, and these estimates may help educate clinicians and support clinical decision making. They cannot be used in isolation, however. Models can estimate the probability of a given outcome for a group of clinically similar patients with a high degree of accuracy, but they cannot reliably predict outcome for individuals.11 At least in the context of deciding whether or not to treat individual patients, it is important to continue to acknowledge, as did physicist Niels Bohr, that “prediction is very difficult, especially about the future.”



View Abstract