For healthcare professionals only

Practice Diagnosis in General Practice

Clinical prediction rules

BMJ 2009; 339 doi: https://doi.org/10.1136/bmj.b2899 (Published 07 August 2009) Cite this as: BMJ 2009;339:b2899
  1. Gavin Falk, clinical research fellow,
  2. Tom Fahey, professor of general practice
  1. 1Department of General Practice, Royal College of Surgeons in Ireland Medical School, Dublin 2, Republic of Ireland
  1. Correspondence to: T Fahey tomfahey{at}rcsi.ie

    In this pair of articles, Gavin Falk and Tom Fahey (doi:10.1136/bmj.b2899) set out what to consider when using a clinical prediction rule, and Dan Mayer (doi:10.1136/bmj.b2901) shows how one such rule, the Ottawa ankle rules, is applied

    What are they?

    Clinical prediction rules quantify the contribution of symptoms, clinical signs, and available diagnostic tests, and stratify patients according to the probability of having a target disorder.1 The outcome of interest can be diverse and be anywhere along the diagnostic, prognostic, and therapeutic spectrum. Developing and validating a clinical prediction rule is a form of observational epidemiological research that requires referring to specific methodological standards.2 3

    These rules usually go through three distinct stages before they are used in a clinical setting:

    • Development of the rule—establishing the independent and combined effect of explanatory variables (or clinical predictors), which can be symptoms, signs, or diagnostic tests

    • Narrow and broad validation—the explanatory variables or clinical predictors in the derivation set are assessed in separate populations

    • Impact analysis—a randomised controlled trial measures the impact of applying the rule in a clinical setting in terms of patient outcome, health professionals’ behaviour, resource use, or any combination of these.

    The CAGE score (box) is a clinical example of a rule developed to aid in the diagnosis of alcohol abuse.

    The CAGE questionnaire

    Each positive answer scores one point.

    • 1. Have you ever felt you should Cut down on your drinking?

    • 2. Have people Annoyed you by criticising your drinking?

    • 3. Have you ever felt bad or Guilty about your drinking?

    • 4. Have you ever had a drink first thing in the morning to steady your nerves or to get rid of a hangover (Eye-opener)?

    When are they used?

    Clinical prediction rules are most commonly used at the stage of refining a diagnosis alongside quantifying the probability of a target disorder (fig 1). Applying a rule often requires a bayesian approach to diagnosis: estimating a clinically likely pretest probability for a target disorder, then applying a likelihood ratio derived from the presence or absence of the clinical features of the rule (similar to applying a test result), which in turn enables a revised estimate of clinical probability.4 Whether a clinician wishes to “rule in” or “rule out” a disorder is likely to be specific to the setting of care and the nature and severity of the target disorder. For instance, clinical prediction rule s may be used in primary care to rule out a disorder, provide reassurance, or adopt a “watchful waiting” strategy. In these instances rules with a high sensitivity and low negative likelihood ratio (ratio of false negative to true negative in patients with a negative test result) are preferred.5 In the same way, ruling in a diagnosis is desirable in secondary care settings where the emphasis is usually on establishing a firm diagnosis and starting appropriate treatment or conducting more expensive and invasive diagnostic tests.5 In these settings rules with a high specificity and high positive likelihood ratio (ratio of true positive to false positive in patients with a positive test results) are preferred.5

    Figure1

    Fig 1 Stages and strategies in arriving at a diagnosis

    This quantitative approach applied to a clinical example—that of alcohol abuse—is shown in figure 2.6 The pretest probability of alcohol abuse in general practice is estimated as about 5%; if the CAGE questionnaire is administered and the patient scores 3, then the positive likelihood ratio is 13.1.6 Applying this positive likelihood ratio to the pretest probability produces a post-test probability of alcohol abuse of 41.3%, which warrants intervention in terms of more detailed assessment, counselling, and management.4 Other examples of clinical prediction rules are provided in the table on bmj.com.

    Figure2

    Fig 2 Calculating post-test probability of alcohol abuse using positive likelihood ratio estimate,6 and equivalent method with nomogram

    How do they go wrong?

    Clinical prediction rules, like diagnostic tests, are subject to biases that affect their validity and application in clinical practice.5 7 Heuristic reasoning (cognitive strategies people learn or adopt when making decisions or solving problems) usually works well when the rule is relatively simple (with few clinical variables) but when the situation is complex, heuristic reasoning may produce errors.

    Common errors when applying rules include:

    • Incorrect estimates of pretest probability of disease (for example availability bias overestimates the probability of vivid or easily recalled events, such as rare but memorable disease)

    • Inaccuracy due to methodological problems in their derivation (spectrum bias occurs when the study population from which the accuracy of a rule is derived has a different clinical spectrum, usually with more severe or advanced disease, than the population of patients to which the rule is applied, and can lead to the sensitivity and specificity of a clinical feature or diagnostic test incorporated in a rule being exaggerated7)

    • Imprecise quantitative estimates in rules (when sample size considerations are not reported clearly 7 8) make the precision of the diagnostic, prognostic, or therapeutic recommendations of a rule less certain.

    An example of the challenges of applying a rule to a primary care setting is the CRB-65 score, which is used to predict 30 day mortality in patients with community acquired pneumonia.9 The score, (representing confusion, respiratory rate, blood pressure, and age over 65) was derived and validated in three cohorts of patients admitted to hospital in the United Kingdom, New Zealand, and the Netherlands.9 Subsequent validation in a separate, community based cohort in the Netherlands showed lower 30 day mortality across all strata of risk than in hospital based patients.10 Low risk patients can be accurately identified with CRB-65, but the optimal referral threshold for a patient with suspected community acquired pneumonia, and how it might affect their subsequent management and survival, is unclear.4 10

    How can we improve?

    Methodological standards concerning the conduct and reporting of clinical prediction rules are well documented.1 2 STARD (Standards for Reporting of Diagnostic Accuracy) is likely to provide a framework for improved conduct and reporting of published rules, particularly in combating spectrum bias and selection bias in the populations of patients used in these studies.11

    Recent initiatives aim to increase sample size substantially and improve precision by means of simplified study protocols and web based recruitment. For example, a study of clinical prediction rules (www.descarte.co.uk) is currently under way in UK primary care, aiming to recruit 18 000 patients with a sore throat in general practice and assess the clinical features that predict further complications.

    Lastly, the accurate recall and implementation of rules can be facilitated by computer based clinical decision support systems (CDSSs) that quantify diagnostic and prognostic information and provide clinicians with patient specific recommendations.12

    Notes

    Cite this as: BMJ 2009;339:b2899

    Footnotes

    • Contributors: Both authors prepared and amended the manuscript together. TF is the guarantor.

    • Funding: GF is supported as a part-time clinical research fellow funded by the HRB Centre for Primary Care Research, HRC/2007/1.

    • Competing interests: None declared.

    • Provenance and peer review: Commissioned; externally peer reviewed.

    References

    View Abstract