Predictive accuracy of the Framingham coronary risk score in British men:prospective cohort study

BMJ 2003; 327 doi: http://dx.doi.org/10.1136/bmj.327.7426.1267 (Published 27 November 2003)
Cite this as: BMJ 2003;327:1267
  1. Peter Brindle, Wellcome training fellow in health services research (peter.brindle{at}bristol.ac.uk)1,
  2. Emberson Jonathan, research statistician2,
  3. Fiona Lampe, lecturer in medical statistics and epidemiology2,
  4. Mary Walker, senior lecturer in epidemiology2,
  5. Peter Whincup, professor of cardiovascular epidemiology3,
  6. Tom Fahey, professor of primary care medicine4,
  7. Shah Ebrahim, professor in epidemiology of ageing1
  1. 1Department of Social Medicine, University of Bristol, Bristol BS8 2PR
  2. 2Department of Primary Care and Population Sciences, Royal Free and University College Medical School, London NW3 2PF
  3. 3Department of Community Health Sciences, St George's Hospital Medical School, London SW17 0RE
  4. 4Tayside Centre for General Practice, University of Dundee, Dundee DD2 4AD
  1. Correspondence to: P Brindle
  • Accepted 16 October 2003

Abstract

Objective To establish the predictive accuracy of the Framingham risk score for coronary heart disease in a representative British population.

Design Prospective cohort study.

Setting 24 towns in the United Kingdom.

Participants 6643 British men aged 40-59 years and free from cardiovascular disease at entry into the British regional heart study.

Main outcome measures Comparison of observed 10 year coronary heart disease mortality and event rates with predicted rates for each individual, using the relevant Framingham risk equation.

Results Of 6643 men, 2.8% (95% confidence interval 2.4% to 3.2%) died from coronary heart disease compared with 4.1% predicted (relative overestimation 47%, P < 0.0001). A fatal or non-fatal coronary heart disease event occurred in 10.2% (9.5% to 10.9%) of the men compared with 16.0% predicted (relative overestimation 57%, P < 0.0001). These relative degrees of overestimation were similar at all levels of coronary heart disease risk, so that overestimation of absolute risk was greatest for those at highest risk. A simple adjustment provided an improved level of accuracy. In a “high risk score” approach, most cases occur in the low risk group. In this case, 84% of the deaths from coronary heart disease and non-fatal events occurred in the 93% of men classified at low risk (< 30% in 10 years) by the Framingham score.

Conclusion Guidelines for the primary prevention of coronary heart disease advocate offering preventive measures to individuals at high risk. Currently recommended risk scoring methods derived from the Framingham study significantly overestimate the absolute coronary risk assigned to individuals in the United Kingdom.

Introduction

Coronary heart disease is a major cause of death and disability in the developed world.1 Identification of people who are at high risk of developing coronary heart disease but currently have no symptoms has become an accepted method for the primary prevention of coronary heart disease in many countries. The national service framework for coronary heart disease in England and Wales states that people whose estimated risk of coronary heart disease based on a specified risk factor profile is ≥30% over 10 years should be identified and offered appropriate advice and treatment.2 European, American, and Canadian guidelines also use predicted 10 year risk to identify people for risk factor modification.36

It is recommended that risk assessment be performed using one of several methods that combine values for different risk factors to produce a quantitative risk estimate.79 These methods use regression equations derived from a population sample of the Framingham heart study and the Framingham offspring study.10 Despite evidence that Framingham risk equations systematically overestimate risk of coronary heart disease in populations with lower coronary heart disease mortality, risk scoring methods based on these equations have been introduced widely.1113 It remains unclear, however, whether the Framingham risk score accurately predicts risk of coronary heart disease in the British population. We assessed the ability of the Framingham risk equations to predict death from coronary heart disease and the combination of fatal and non-fatal coronary heart disease events that is the outcome used in current scoring methods, in a representative population of British men over a 10 year period.79

Participants and methods

The Framingham studies

The risk assessment methods recommended for British and European use are adapted from published equations derived from 5573 men and women from the Framingham heart study and the Framingham offspring study. People aged 30-74 and free of cardiovascular disease were included, and risk estimates for cardiovascular diseases were derived from around 12 years of follow up. Equations were derived for six outcomes, two of which we consider here: death from coronary heart disease, and all fatal and non-fatal coronary heart disease events (box).10 14

The British regional heart study

The British regional heart study is a prospective study of 7735 men, aged 40-59 years at entry (1978-80), who were randomly selected from the age and sex registers of one general practice in each of 24 towns in the United Kingdom. The towns were selected to represent the range of cardiovascular disease mortality in the United Kingdom at the time.15 The response rate was 78%, and participants have been followed up for cause specific mortality using the NHS central registers and for cardiovascular morbidity through regular two yearly reviews of general practice records, with fewer than 1% of participants lost to follow up.16 For the purpose of our analysis, we chose the criteria used to define pre-existing cardiovascular disease in the British regional heart study to match those of the Framingham study as closely as possible (table 1).17

View this table:
Table 1

Risk factor and definitions of end points

Framingham risk equations for coronary heart disease death (B1) and coronary heart disease events (B2) in men over 10 years

Step 1

For coronary heart disease mortality calculate

μ = 11.2889 - 0.588xlog(systolic blood pressure) - 0.1367xsmoking - 0.3448xlog(total/high density lipoprotein cholesterol) - 0.1237xelectrocardiographic left ventricular hypertrophy - 0.944xlog(age) - 0.0474xdiabetes

σ= exp(2.9851 - 0.9142μ) (B1)

For coronary heart disease events calculate

μ = 15.5303 - 0.9119xlog(systolic blood pressure) - 0.2767xsmoking - 0.7181xlog(total/high density lipoprotein cholesterol) - 0.5865xelectrocardiographic left ventricular hypertrophy - 1.4792xlog(age) - 0.1759xdiabetes

σ = exp(- 0.3155 - 0.2784x(μ - 4.4181)) (B2)

Step 2

For both equations calculate:

μ = (log(10) - μ)/σ Length of follow up = 10 years

Step 3

The predicted probability is then given by:

p=1 - exp(-exp(u))

*Variables smoking, electrocardiographic left ventricular hypertrophy, and diabetes are set to 1 when present and 0 when absent. Systolic blood pressure measured in mm Hg and age in years

Statistical methods

Assessing the accuracy of the Framingham equation

Using the appropriate Framingham equations, we calculated the risk of death from coronary heart disease and all coronary heart disease events over a 10 year period for each of the men in the British regional heart study who were initially free of cardiovascular disease and had complete information on risk factors (see box). We categorised the men into groups defined by quintiles of Framingham risk, systolic blood pressure, total to high density lipoprotein cholesterol ratio, and age. We compared the average predicted event rates within each quintile for both end points with the observed 10 year rates. The Hosmer Lemeshow test was used to assess goodness of fit.18

Geographical variation

To assess any regional differences between observed and predicted rates, we also categorised the men by region of residence at baseline: Scotland, the north of England, the Midlands and Wales, and the south of England.

Discrimination

To assess the performance of the screening test at identifying individuals at “high risk,” we calculated the sensitivity and specificity for risk score thresholds of ≥30% and ≥15% over 10 years.

Results

Of 7735 men recruited to the British heart regional study, 6942 (89.7%) were free of definite angina on the Rose angina questionnaire and had no recall of a doctor diagnosis of cardiovascular disease and no electrocardiographic evidence of definite myocardial infarction. Of these men, 6643 (95.7%) had complete data on risk factors at baseline. Table 2 compares the baseline characteristics of these men with those of the 2590 men from the Framingham cohorts used in the derivation of the risk equations.

View this table:
Table 2

Baseline characteristics of men in Framingham studies and British regional heart study without pre-existing cardiovascular disease and with complete data on risk factors

Observed and predicted coronary heart disease mortality

When the coronary heart disease mortality equation (equation B1 in box) was applied to each of the men in the British regional heart study, the predicted number of deaths from coronary heart disease within 10 years was 270 (4.1%). This compared with an observed 183 deaths from coronary heart disease, giving a rate of 2.8% (95% confidence interval 2.4% to 3.2%) over the first 10 years of follow up. Figure 1 displays predicted and observed mortality from coronary heart disease across a range of risk levels (according to the quintiles of Framingham risk, systolic blood pressure, total to high density lipoprotein cholesterol, and age). This relative over-prediction of mortality risk by 47% (P value for goodness of fit < 0.0001) was similar at all risk levels (fig 1a), so that over-prediction of absolute risk was greatest for people at highest risk. Similarly, figures 1b-d show that significant overestimation of risk occurs at all levels of the risk factors concerned, apart from the lowest level of systolic blood pressure. For the combined outcome of fatal coronary heart disease and any diagnosis of myocardial infarction or angina (equation B2 in box), the observed number of events over 10 years was 677 (event rate 10.2%, 95% confidence interval 9.5% to 10.9%) compared with a predicted 1062 (16.0%)–a relative over-prediction of 57% (P value for goodness of fit < 0.0001; fig 2).

Fig 1

Ten year predicted versus observed coronary heart disease mortality with 95% confidence intervals by quintile of Framingham risk, systolic blood pressure, total to high density lipoprotein cholesterol ratio, and age

Fig 2

Ten year predicted versus observed coronary heart disease event rates with 95% confidence intervals by quintile of Framingham risk, systolic blood pressure, total to high density lipoprotein cholesterol ratio, and age

Geographical variation

Table 3 shows the observed versus predicted rates of coronary heart disease mortality and all coronary events by region. Over-prediction by the Framingham equations occurred in all regions but was greatest in the south of England and the Midlands and Wales where there was relatively lower mortality and morbidity than in Scotland and the north of England.

View this table:
Table 3

Ten year predicted versus observed rates of coronary heart disease mortality and all coronary events by region

Recalibration

As the relative over-prediction was about constant at all levels of risk, it was possible for us to adjust the Framingham scores by dividing the calculated score for each individual by the amount of over-prediction. Recalibrated probabilities of death from coronary heart disease were therefore obtained from the 10 year predictions by dividing the final score by 1.47. For example, an individual predicted to have a 5% chance of a fatal coronary heart disease event within 10 years had a recalibrated risk of 3.4%. After making this correction, the predicted risk became close to the observed rate at all levels of risk (fig 3a), as indicated by a substantial decrease in the χ2 statistic for goodness of fit from 30.2 to 3.4. Similarly, the risk equation for coronary heart disease events was corrected to take into account the 57% relative over-prediction by dividing the Framingham prediction by 1.57 (fig 3b). Again a large decrease was observed in the goodness of fit statistic from 155.3 to 24.6.

Fig 3

Predicted coronary heart disease death and coronary heart disease event risks before and after recalibration with observed 10 year rates

Discrimination

When we applied the coronary event equation (see box) to the baseline data in the British regional heart study, 444 men (6.7%) had a predicted 10 year coronary heart disease event risk of ≥30% (average predicted risk 36.2%), of whom only 106 (out of the 677 men with a coronary heart disease event) actually had a coronary heart disease event within the following 10 years–a sensitivity of 16% (106/677). The sensitivity increased to 75% (509/677) when a 15% risk threshold was used, but this was at the expense of a large drop in specificity from 94% (5628/5966) to 55% (3258/5966) and a large increase in the proportion of men classified as high risk (from 6.7% to 48.4%). Similar estimates of sensitivity and specificity were obtained when using these thresholds to identify individuals at high risk of coronary heart disease death within 10 years.

When the recalibrated equation was used, those in the high risk group identified by using the ≥30% threshold would now constitute only 0.5% of the population and identify only 1.8% (12/677) of the coronary heart disease events occurring within 10 years, so that preventive interventions restricted to this group would have a limited population impact. If a ≥15% threshold was used with the recalibrated equation, 17% of the population would be classified as high risk, and 37% (249/677) of coronary heart disease events would be identified. The specificities at the 30% or more and 15% or more thresholds using the recalibrated equation would be 99.6% (5944/5966)and 85% (5055/5966), respectively.

Discussion

The Framingham equations used in current risk scoring methods over-predict the risk of mortality from coronary heart disease and all fatal and non-fatal coronary heart disease events by 47% and 57%, respectively, compared with observed events in a representative sample of British men. The relative degree of over-prediction was similar at all levels of individual risk.

Limitations of study

The Framingham study included the category “unrecognised myocardial infarction” in the ascertainment of non-fatal coronary heart disease events, therefore a potential source of bias could exist. However, this seems to have had little effect, as both fatal and all events were similarly overestimated. This is probably because the definition of a coronary heart disease event in the British regional heart study was broad, including all possible cases of myocardial infarction and angina documented in the medical records. Although coronary heart disease death is a more accurately defined end point, there is still a possible source of bias in the way the cause of death was identified. The British regional heart study used death certificates and postmortem reports, whereas in the Framingham study the cause of death documented on the certificate was verified by reviewing autopsy data, hospital records, and records of the attending doctor. However, coronary heart disease is an over-reported cause of death on death certificates, so any bias would tend to result in our analyses being conservative, underestimating the level of over-prediction.19 Some minor differences are apparent in the individuals excluded from both studies and in the use of a more sensitive Framingham definition of diabetes. The numbers of patients with diabetes, however, are small, and because the unidentified predicted risk for such patients would be underestimated, any bias would again lead to our results being conservative. A further limitation of our study is that its conclusions cannot be assumed to apply to women, although the effect of altering risk factors and the accuracy of models predicting coronary heart disease are similar in both men and women and therefore the findings are likely to be relevant to risk prediction in women.12 20

Other studies

The Whickham (UK) study, conducted between 1972 and 1974 with a single follow up 20 years later, compared observed events in 1700 men and women with rates predicted by a Framingham equation.21 22 The observed and predicted event rates in the higher risk population (coronary heart disease event rate greater than 1.5% per year) were similar, but the Framingham equation underestimated risk in those at lower risk. In the Whickham study, the annual coronary heart disease event rate was 1.56% compared with 1.02% for men in the British regional heart study. This may reflect the true risk of the population from the north east of England, or might be because the ascertainment of coronary heart disease events was broad, including all participants who had had minor electrocardiographic changes on follow up and all deaths with any mention of ischaemic heart disease.

A trial investigating the effectiveness of pravastatin in the primary prevention of coronary heart disease found a close agreement between the observed coronary heart disease event rate in the placebo group over 4.4 years and the value predicted from the Framingham equation. Both the Scottish location and the other inclusion criteria of the trial, however, led to the inclusion of a group at particularly high absolute risk, with an annual coronary heart disease event rate in the placebo group of 1.59%.23 In German, Italian, and Danish studies, Framingham risk scores with differing outcomes have been shown to overestimate risk by up to 50%.12 13 24 A European based risk score has been devised to address this.25 In a comparison of the British regional heart study, the prospective cardiovascular Munster (PROCAM) heart study, Dundee, and Framingham risk functions, no direct validation was possible because different end points were used and no follow up data were collected.26

Explanations for different predicted and observed risk

The over-prediction of 10 year risk by the Framingham equations in our analysis is likely to reflect a true difference in the levels of risk between the two populations and is unlikely to be due to over fitting of the Framingham data, as the number of risk factors considered is modest compared with the number of events observed.27 Coronary heart disease mortality in England and Wales in 1980 was 30% lower than that in the United States in 1970.28 The difference between this figure and the 47% overestimation found in our study may be due to differences between national statistics and the study populations as well as the predictive inaccuracy of the Framingham equation in the population participating in the British regional heart study. Furthermore, possible explanations for the differences in US and British mortality may be better control of coronary risk factors and better treatment of coronary heart disease experienced by the later British cohort.

Risk functions derived from the Framingham study and others are reasonably consistent at ranking individuals according to their relative risk, and differences between the observed and predicted risk usually depend on the background risk of the population to which the function is applied.12 24 We have shown inter-regional variation suggesting that over-prediction is greater in the regions of Britain with lower background coronary heart disease risk. Differences in observed and predicted risk have been shown in different countries and between ethnic groups and may be attributable to other risk factors that are not included in the model.12 24 29

Implications

Overestimation of an individual's true risk and the poor sensitivity of the recommended tool to identify and target individuals for treatment have important implications for a national screening test.2 An overestimated assessment of coronary heart disease risk will undermine a patient's ability to make an informed choice about starting preventive treatment, may cause unnecessary anxiety, and may affect life insurance premiums.30 If the patient's absolute risk is lower than predicted, the absolute benefits of intervention will be smaller and the balance of risks and benefits less favourable. Additionally, overestimation of risk prediction will adversely affect direct prescribing costs as well as the costs of drug monitoring and dealing with side effects.

The accuracy of risk estimates derived from cohort studies or from randomised controlled trials are always open to the criticism of being out of date compared with current morbidity rates, owing to the delay between the collection of baseline data and the reporting of incident events.31 If coronary heart disease rates continue to fall, the discrepancy between predicted and actual risk is likely to increase, as the decline is not entirely attributable to falls in the risk factors included in the Framingham equation.32 Furthermore, fewer people will fall into the high risk group, causing the proportion of coronary heart disease events prevented by targeting only these individuals to be reduced.

We have shown that current risk scoring methods seem to overestimate coronary heart disease risk and that a simple adjustment can improve their predictive accuracy in the British population. Nevertheless, further refinements are necessary before the substantial variations in coronary heart disease risk found between different regions and different ethnic groups, socioeconomic status, and family history of coronary heart disease can be accommodated into an accurate and effective treatment decision aid.

What is already known on this topic

Primary prevention of coronary heart disease involves identifying patients at high risk and offering them lifelong preventive treatment

Most risk assessment methods rely on equations derived from the Framingham study

Evidence is conflicting as to the suitability of these equations for British and other European populations

What this study adds

Recommended risk scoring methods overestimated coronary risk in a representative British male population

This was similar at all levels of coronary heart disease risk and could be reduced by a simple adjustment

Use of a predicted 30% coronary heart disease 10 year event rate threshold to identify patients at high risk can fail to identify most who go on to have a coronary heart disease event over the following 10 years

Editorial by Hense

Baseline serum total cholesterol and high density lipoprotein cholesterol were measured at the Wolfson Research Laboratories, Birmingham. The British regional heart study is a British Heart Foundation Research Group with additional support from the Department of Health, England. Margaret May provided helpful comments on an early draft of the paper.

Footnotes

  • Contributors SE and TF developed the original idea for this study. PB drafted the paper and JE performed the analyses assisted by FL. PW and MW contributed to the design and execution of the British regional heart study. All authors contributed to the interpretation of data and the writing of the paper and have seen and approved the final version. PB and JE will act as guarantors for the paper.

  • Funding PB is funded by the Wellcome Trust. The views expressed here are those of the authors and not necessarily those of the funding agencies. The funding agencies had no role in the data collection or in the writing of this paper. The guarantors accept full responsibility for the conduct of the study, had access to the data, and controlled the decision to publish.

  • Competing interests None declared.

  • Ethical approval None required.

    bmj.com 2003;327:1267

References

THIS WEEK'S POLL