The risks in risk predictionBMJ 2012; 344 doi: https://doi.org/10.1136/bmj.e4215 (Published 21 June 2012) Cite this as: BMJ 2012;344:e4215
- Catherine McGorrian, cardiologist and UCD Newman scholar,
- Gavin J Blake, consultant cardiologist and UCD senior lecturer
Comprehensive assessment of the risk of cardiovascular disease using a multiple risk factor system is now widely accepted as the method of choice for targeting interventions in primary prevention. However, several risk equations are available, and there is no consensus on which system or score to use.1 In the linked paper (doi:10.1136/bmj.e4181),2 a risk estimation system (QRISK2-2011), which was derived from the UK QRESEARCH database, is examined in an independent UK population to assess its predictive ability. Such external validation studies are necessary in the development of risk estimation systems to prove the accuracy and generalisability of such systems.
This important study shows that QRISK2-2011 has both better callibration (the degree to which the number of events predicted by the risk estimation system agree with the number of events observed) and better discrimination (a measure of how correctly the system ranks risk between individuals) than the National Institute for Health and Clinical Excellence (NICE) version of the Framingham equation. The NICE Framingham equation is based on the 1991 Anderson Framingham equation, which is known to overpredict disease levels in some populations.1 3 Comparisons of the relative validity of different risk estimation tools are often undertaken and can be susceptible to bias.4 Collins and Altman, however, provide a statistically rigorous comparison, and their findings represent an advance in risk prediction for practitioners based in the United Kingdom.
Nonetheless some caveats remain. Both the derivation and validation of risk estimation tools bring substantial methodological challenges. Here, the QRISK validation has been performed in the THIN (the Health Improvement Network) dataset. Traditional cohort studies are expensive and take many years to perform, whereas THIN represents a new type of epidemiological data collection. In THIN, general practice software gathers clinical data to populate a database that can be used for statistics and research. The benefits of such a process are easily appreciated, and the sample size available in THIN is considerable. However, unlike traditional cohort studies, definitions used for the clinical endpoints may vary in such datasets, and although studies of data validity in THIN have been performed,5 these have been less stringent than the case validation procedures undertaken in cohort studies. In addition, there are many missing data in THIN, which the authors deal with by using multiple imputation methods. However, cholesterol concentrations were not known for 78% of patients in THIN, and this reduces the face validity of the risk estimate. Given that statins are so widely used to reduce risk of cardiovascular disease, knowledge of an individual patient’s lipid concentrations would be essential. The generalisability of QRISK to non-UK populations also remains to be shown.
Model parsimony is the concept that risk estimation models and scores should achieve an optimum balance between attaining the best risk estimate and providing a simple and concise score for clinical use. A risk score that has multiple data items, or items that are difficult to collect, is less attractive to busy doctors.1 6 QRISK2-2011 requires the clinician to ascertain 13 different clinic variables, compared with the seven variables in the NICE Framingham equation.2 It may be that a more pared down version of QRISK might attain similar validity results and be easier to apply in clinical practice. Notably, from this external validation, it does not seem that the 2011 version of QRISK is substantially more effective that the 2008 version. Clinicians may call for the inclusion of further risk factors such as diet or physical exercise variables, but previous work on the INTERHEART modifiable risk score has shown that inclusion of further, less powerful, risk predictors does not necessarily improve score discrimination.7 A strong statistical association needs to exist between a risk factor and a disease for that factor to contribute as part of a screening test.8
Collins and Altman show the effects of the QRISK and Framingham tools at different thresholds of risk, thus illustrating perfectly one of the challenges of risk estimation: what level of risk uncertainty can we tolerate, and at what point do we deem that CVD risk become “too high”? From table 4,2 it can be clearly seen that if we choose an arbitrary risk cut-off point in QRISK2-2011 of 15% or more, we will be instituting preventive treatment in more than 10% of women and almost 20% of men in this primary prevention group. Do we err on the side of expanding the population eligible for treatment, such as described by the Cholesterol Treatment Trialists,9 and if so, what are the pharmacoeconomic implications of the potential onslaught of prescribing? Furthermore, the “real world” use of risk estimation tools in primary prevention care is not known. Some doctors may think that they can estimate risk without recourse to a scoring system,10 and there is evidence that prescribing in primary prevention may be more ad hoc than evidence based.11
Thus risk estimation remains an imperfect science. It may be that using existing tools in an open consultation with our patients will lead to better risk factor modification. Concepts such as risk age, explored in tools such as SCORE12 and QRISK2-2012 (http://qrisk.org/), provide a more tangible description of risk for patients, as do features such as showing patients the relative contributions of adverse risk factors to their personal risk profile. QRISK represents an improvement in risk estimation for UK practitioners, but its generalisability to other populations is unclear and the problem remains as to what represents an “acceptable” versus “unacceptable” risk level.
Cite this as: BMJ 2012;344:e4215
Competing interests: Both authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Provenance and peer review: Commissioned; not externally peer reviewed.
Log in using your username and password
Log in through your institution
Register for a free trial to thebmj.com to receive unlimited access to all content on thebmj.com for 14 days.
Sign up for a free trial