Risk score would be more helpful

BMJ 1995; 311 doi: (Published 25 November 1995) Cite this as: BMJ 1995;311:1398
  1. Gordon Murray, director
  1. Robertson Centre for Biostatistics, Boyd Orr Building, University of Glasgow, Glasgow G12 8QQ

    Voss and colleagues describe data which are of considerable interest, and yet I find their paper frustrating in that the analysis and presentation of their data stops just short of what is required to assess the clinical relevance of their findings. Moreover, their conclusions are speculative and go beyond what can be supported by the data as presented in their paper.

    The problem being tackled is inherently multivariate (what can be said about an individual patient with a certain set of features and symptoms?) as is the authors' primary analysis, and yet the quantitative presentations of results are all univariate (tables II, III, VI, and VII). The data as presented do not allow one to deduce how many patients had, for example, at least one risk factor for an abnormality on computed tomography, nor can one say anything about the odds ratios for a patient with a certain combination of risk factors. Thus the data give little assistance in the making of management decisions concerning individual patients, and neither do they allow one to assess the properties of a strategy such as “admit and urgently scan all patients with a skull fracture or some objective evidence of intracranial problems, or both.”

    Both of these issues could have been dealt with very simply. For decision making relating to individual patients the logistic regression analyses could have been reported by giving a risk score which could be translated, for an individual patient, into a probability of having an abnormality on computed tomography or requiring neurosurgical intervention. In terms of assessing a management strategy the authors could have given the numbers of patients flagged as being “at risk” together with the sensitivity and specificity for detecting problems. Several different strategies could have been presented and their performance compared. It would be interesting, for example, to see whether the inclusion of speech disturbance in a decision rule was useful. This feature was significant in both sets of univariate analyses, and yet it seems unlikely that a feature which is present in only 1% of reattenders would be of value in any decision making process.

    The estimates of sensitivity and specificity for any such rules would be optimistic because of the circularity of assessing the performance of a rule on the same data set as was used to derive the rule. This would be a particularly severe problem in terms of predicting the requirement for a neurosurgical operation because of the small number of such procedures in the study population (30). A further prospectively collected data set would be required to give a reliable assessment of performance, but a circular assessment would at least give some indication of whether the features identified by the authors might allow more rational decisions to be made in this difficult clinical area.

    Authors' response

    Our basic aims in this paper were twofold: firstly, to point out that “comeback” patients are a high risk group in themselves and, secondly, to identify features which might suggest that immediate further investigation or referral should be undertaken. It was intended to be exploratory and descriptive and thus was not formulated as a test of any particular strategy. The idea of presenting the data in a univariate manner was to emphasise the increased risk associated with the presence of certain clinical features in our patients as compared with those who did not have these features. The list of these features was obtained by multivariate analysis but clearly needed separation to allow the relative risk of an adverse outcome to be assessed. We think that the figures as presented give a fairly clear picture to the non-statistician and at the same time have a clear message for the clinician, which is “when confronted by a comeback patient with any high risk feature the prudent action is to arrange urgent referral.”

    We think that the development of yet another risk score would not contribute understanding to the care of the head injured patient, and we are concerned that it may lead to undue reliance on such a score. We think we have made it clear enough that ideally all reattenders should undergo computed tomography but that if this is not practicable then those with any of the risk factors or associated features we identified should be given priority for referral. If “soft” features are persistent one would be most unwise to ignore them, and thus we don't think an algorithm or risk score are more helpful than general clinical advice. The difficulty and validity of developing sensitive and specific rules retrospectively from data which have already been collected is emphasised by the commentator, and we would prefer to leave this task to future researchers to carry out prospectively.

    View Abstract