Introduction

Top Abstract Introduction Methods Results Discussion References

Cardiovascular disease is the most important cause of death in the world today,¹ with a lifetime risk at age 40 of 1 in 2 for men and 1 in 3 for women in developed countries.² In the United Kingdom, 66 000 people aged under 75 die from cardiovascular disease each year.³ The British government has set a target of a reduction in cardiovascular disease in this age group of at least two fifths by 2010. Prevention on this scale requires major action to reduce the adverse consequences of risk factors for cardiovascular disease. ^{4 5} Specific interventions will need to be used for hypercholesterolaemia, ^{6 7} hypertension, ^{8 9} smoking,¹⁰ and diabetes ^{11 12} (primary prevention strategies), and people already suffering from cardiovascular disease will need to be treated (secondary prevention).^13-15

People with the highest risk of coronary heart disease stand to gain the most, in absolute terms, from interventions, and effective methods to identify such people are required. To date, most emphasis has been placed on secondary prevention; those most likely to benefit from secondary prevention are identified relatively easily on the basis of a history of cardiovascular disease. Identification of patients suitable for primary prevention is less straightforward, and recent guidelines have recommended the use of formal risk scoring with an algorithm based on the Framingham equation to identify high risk patients.^16-19 The recent national service framework for coronary heart disease in the United Kingdom will eventually require primary prevention of coronary heart disease on the basis of risk calculations for coronary heart disease.²⁰ Although the Framingham equation has been shown to predict events when it was applied retrospectively to placebo populations recruited to statin trials,²¹ evidence for the use of risk calculations in primary care is conflicting.

General practitioners and practice nurses are able to use risk tables accurately when given patient data in the form of scenarios.²² However, accuracy was only moderate in a study that compared risks calculated in daily practice by general practitioners and practice nurses using the New Zealand risk tables with a reference calculation of risk.²³

An alternative to calculation is for clinicians to use their knowledge of the patient to estimate the risk subjectively. This method is quick and does not need a full set of risk factors to provide an answerbut studies that evaluated this approach showed disappointing results. A Canadian study found that doctors accurately estimated the relative risk for hypothetical patients, but that they systematically overestimated the absolute risk of developing coronary heart disease.²⁴ Another study, in the United Kingdom, compared subjective estimates of risk with calculations made using the Framingham equation and found that general practitioners consistently underestimated the risk.²⁵

We are not aware of any study that directly compared subjective estimates of risk with formal calculations on the basis of data available routinely in patients' records. This study aimed to assess the accuracy with which general practitioners and practice nurses apportion the risk of coronary heart disease, in a quasi-routine clinical setting, by estimating and calculating, with four different calculation tools, the risk of coronary heart disease, by using a randomly selected sample of patient records, quota sampled for varied risk factors for cardiovascular disease.

	Methods

Top Abstract Introduction Methods Results Discussion References

We recruited 18 practices from the Midlands Research Practices Consortiuma research network that comprises more than 70 general practices scattered across the West Midlands. The network's overall practice populations are generalisable, in terms of socioeconomic gradient and type of location, to England and Wales.

Each practice performed computerised searches to identify patients in five groups at risk of coronary heart disease: those with hypertension, type 1 or 2 diabetes or hypercholesterolaemia; those who were current smokers; and those with current coronary heart disease (defined as history of angina, previous myocardial infarction, or previous coronary artery bypass or coronary angioplasty). We asked practices to use their own disease registers in the searches, because this represents the way that groups at risk of coronary heart disease would be identified by risk scoring in routine practice. Once the five groups had been identified, we randomly selected 10 sets of notes by choosing two patients from each group. Quota sampling was used to ensure that the sample contained no fewer than three women and no fewer than three patients over 65 years of age.

Four different risk calculation tools, representing the most widely used tools and a spectrum of different formats (risk chart or electronic calculator), were selected for comparison. The tools were the Sheffield table (a paper chart),²⁶ the New Zealand hypertension guideline risk table (a paper chart),¹⁹ the joint European societies' recommendations on prevention of coronary heart disease (a wall chart),²⁷ and the joint British societies' recommendations on the prevention of coronary heart disease in clinical practice (a computer program, which is also available as a paper chart).²⁸ Each tool uses slightly different data to calculate risk (see table 1).

In each practice, a general practitioner and a practice nurse estimated and calculated risks separately and blinded to each other in a dedicated non-clinical session (to approximate to an audit session in the practice). The practitioners were asked first to subjectively estimate the coronary heart disease risk for the 10 chosen patients, using only information in the patients' notes and computer records. They were then asked to calculate the risk, using the four different calculation tools, from the information available in the patient's records. To prevent an ordering effect, each practice used the calculation tools in a different predetermined sequence. Instructions about how to use the calculation tools were as given in the first published descriptions of the tools. ^{19 26-28} For each patient, the general practitioner or practice nurse recorded the risk calculation (or explained why they were not able to calculate it).

Analysis
The risks obtained with each of the tools were divided into high or low risk. The recommendations in the national service framework for coronary heart disease define high risk as a >30% risk of coronary heart disease at 10 years, and they say that interventions, such as statins, should be used in primary prevention for patients with risks higher than this. The nearest equivalent to this level provided by each table was chosen as the cut-off value: >3% annual risk of coronary heart disease for the Sheffield table, >20% risk of cardiovascular disease at five years for the New Zealand table (note the apparently higher risk band used because cardiovascular risk rather than coronary heart disease risk was calculated), >20% risk of coronary heart disease at 10 years for the European chart, and >30% risk of coronary heart disease at 10 years for the British program.

All estimates and calculations were compared with one prospectively derived "reference standard" score of risk, which was calculated for each patient by using a computerised version of the original Framingham equation¹⁶ and data independently collected from the patients' records by researchers. Where datafor example, concentrations of total cholesterol or high density lipoprotein (HDL) cholesterolwere missing, age and sex adjusted national averages were used in the reference calculation.²⁹ In line with current British recommendations, the reference standards were classified as high risk if the risk of coronary heart disease at 10 years was greater than 30% or if the patient had pre-existing cardiovascular disease.²⁰

We performed analyses with SPSS for Windows (version 10). We compared estimates and calculations of risk with reference calculations by using the K statistic to test agreement above chance,³⁰ and estimates made by general practitioners and practice nurses for the same patient were also compared. We separately examined risks calculated for patients with pre-existing cardiovascular disease to find out in how many scores for the risk of coronary heart disease had been calculated inappropriately. We determined the sensitivity and specificity of each method compared with the reference calculation.

	Results

Top Abstract Introduction Methods Results Discussion References

Availability of relevant data in routine patient records
The records of 70 patients contained evidence of cardiovascular disease; these patients could be classified as being at high risk of coronary heart disease without risk estimation or calculation. Risk calculation was appropriate for the remaining 110 patientssex and age were documented in all cases. Blood pressure and smoking status were recorded for most patients, although cholesterol concentrations, particularly of HDL cholesterol, were present in fewer records (table 1). The number of patients who could be classified from these data with each calculation tool varied from 92 (51%) for the British program and New Zealand table to 161 (89%) for the Sheffield table (table 1).

Subjective risk estimation and formal risk scoring
Subjective estimates by practice nurses agreed with the reference calculation in 125 (69%, =0.38) cases and with general practitioner estimates in 131 (71%, =0.45) cases (table 2). The number of formal (objective) risk calculations attempted and the proportion correct varied (table 2). Nurses using the British program achieved the closest agreement with the reference calculation (=0.65), but attempted the fewest calculations with this tool (96, 53%). General practitioners achieved the closest agreement with the reference calculation when using the British program (=0.55), the New Zealand table (=0.53), and the Sheffield table (=0.51), but they attempted fewer calculations with the former (99 v 134 v 131). The greatest number of calculations was attempted with the European table (148, 82% by practice nurses and 149, 83% by general practitioners), but little agreement with the reference calculation was seen (=0.38 for practice nurses, 0.33 for general practitioners).

Some risk calculations were completed for people for whom risk scoring was inappropriatefor example, patients with a history of coronary heart disease, who automatically are eligible for secondary prevention (table 3). Partly because of this systematic error in scoring, the overall trend using objective risk scoring was towards an underestimation of risk (table 2).

Repeatability of subjective estimates and formal risk calculationscomparing results from general practitioners and practice nurses for the same patientwas moderate (between =0.47 for estimates and =0.58 for scoring risk with the British program) (table 4).

Sensitivity was lowest for practice nurses using the Sheffield table to calculate risk (58%) and highest for general practitioners using the British program (80%) (table 5). Specificity was lowest for practice nurses using the New Zealand table (63%) and highest for practice nurses using the British program (100%).

	Discussion

Top Abstract Introduction Methods Results Discussion References

In this study, general practitioners and practice nurses were asked to assess whether people were at high or low risk of coronary heart disease, consistent with the recommendations of the national service framework for coronary heart disease. The greatest overall accuracy was achieved when primary care clinicians used the British program to calculate coronary heart disease risk.

Limitations of formally calculating risk
An important limitation of using formal calculations to assess the risk of coronary heart disease in primary care is that information necessary for determining risk factors may not be available. This is a particular problem for the New Zealand table and British program, which use concentrations of HDL cholesterol (the notes of only a fifth of patients (23/110) contained this information when patients with existing coronary heart disease were excluded). In contrast, it was possible to calculate risk for most patients with the Sheffield and European tools.

A second problem is that the calculations may be used inappropriatelyrisk scores were calculated in up to 40% of patients with coronary heart disease, even though such patients can be classified as high risk because of their disease history alone. This tended to be less of a problem with the British program, perhaps because the instructions about not applying the program to such patients were clearer.

Accuracy of tools
When enough data are available, general practitioners and practice nurses can achieve reasonable accuracy with the different calculation tools. The British program was the tool that was used most reliablyit gave the best results in terms of sensitivity for general practitioners and practice nurses and specificity for practice nurses. With the possible exception of the British program, risk calculators were no more accurate than subjective assessment for determining the risk of coronary heart disease. This is likely to be because the calculators were used in inappropriate patients and because of the poor availability of data needed for calculations.

For the New Zealand table and the British program, risk was calculated for several patients in whom data about concentrations of HDL cholesterol were not available. Although it should only have been possible to use the table to calculate risk in 22 out of 110 patients without known coronary heart disease, nurses and general practitioners gave results for 78 patients and 64 patients, respectively. In most of these cases, the clinician seems to have assumed that the concentration of HDL cholesterol was 1 mmol/l.

Accuracy of tools in identifying patients at high or low risk
Assessing whether people are at high or low risk of coronary heart disease is the most clinically relevant way of using the risk calculation tools, because a dichotomous decision has to be made as to whether or not to treat each patient. Cut-off values closest to 30% risk at 10 years were chosen for each calculation tool; this will have resulted in some variation in the results from the different tools and the reference standard since, with the exception of the British program, the cut-off values were slightly different from this level. As expected, a higher threshold (such as the 3% per year risk used in the Sheffield table) gave a less sensitive but more specific calculation, but a lower threshold (such as the 20% 10 year risk used in the European chart) did not give the expected more sensitive and less specific assessments. It is not clear why; perhaps the choice of >20% 10 year risk as the cut-off value for the European chart (necessary because of the categories used in that chart) may have led to its relatively poor performance.

Accuracy of estimates and calculations
Past studies have not compared the results of subjective estimates with calculations in the same patients. Grover et al asked 253 general practitioners attending an educational meeting to determine risk for four theoretical patients.²⁴ The general practitioners systematically overestimated the absolute baseline risk of developing coronary disease and the risk reductions associated with specific interventions. Montgomery et al compared absolute risk estimations for 397 patients with hypertension with risk scores calculated with the Framingham equation.²⁵ Estimates were correct in 21% of patients; 63% of risks were underestimated and 16% overestimated. Neither study analysed the results in terms of the choices between high and low risk and whether to initiate formal preventive treatment or not, which is the most important decision for primary care.

Accurate use of tools
Few studies have looked at how accurately professionals use tables to calculate risk. In a Scottish study that used case histories to test how accurately general practitioners and practice nurses used the Sheffield and New Zealand tables and the British program, high and low risk thresholds were used, and enough data were provided to make all calculations possible; the table and program performed best in this study, producing calculations that were correct in most cases.²² Peters et al found only moderate agreement between the reference standard and risk scores calculated by 34 general practitioners and five practice nurses who used the New Zealand table for 228 hypertensive patients (weighted =0.56).²³

Limitations of our study
The main limitations of our study are that the general practitioners and practice nurses performed the estimates and calculations in protected time outside of consultations, and that they were all drawn from practices with an interest in research. Results achieved within the time pressures of a consultation would be expected to be worse than those in our study.

The lack of information (especially about concentrations of HDL cholesterol) in the patient records meant that we had to use estimates to calculate the reference standard, as has been the case in past studies.²³ Although such estimates may introduce some error, the reference standard will still represent the best evaluation of risk that is possible, given what is known about the patient.

Implications for clinical practice
The present study did not directly measure the accuracy with which general practitioners and practice nurses use a particular table. However, the results suggest that problems with extracting data from records and applying the tools only to appropriate patients may be at least as important as simple errors in calculations.

Adequate training is required to ensure that the risk calculation tools are used only for appropriate patients. The availability of data about risk factors needed for these tools, especially concentrations of total cholesterol and HDL cholesterol, needs to be improveda point emphasised in the national service framework for coronary heart disease. In the absence of complete data, subjective estimates of risk seem to be a reasonable alternative.