Intended for healthcare professionals

CCBYNC Open access
Research

Prediction model to estimate presence of coronary artery disease: retrospective pooled analysis of existing cohorts

BMJ 2012; 344 doi: https://doi.org/10.1136/bmj.e3485 (Published 12 June 2012) Cite this as: BMJ 2012;344:e3485
  1. Tessa S S Genders, clinical epidemiologist12,
  2. Ewout W Steyerberg, professor of medical decision making3,
  3. M G Myriam Hunink, professor of radiology and clinical epidemiology (Erasmus) and adjunct professor of health decision sciences (Harvard)1227,
  4. Koen Nieman, cardiologist, medical coordinator of ICCU, and assistant professor24,
  5. Tjebbe W Galema, cardiologist4,
  6. Nico R Mollet, staff radiologist24,
  7. Pim J de Feyter, professor of non-invasive cardiac imaging24,
  8. Gabriel P Krestin, professor and chairman of department of radiology2,
  9. Hatem Alkadhi, senior radiologist5,
  10. Sebastian Leschka, associate professor, senior staff radiologist, section head of computed tomography, and section head of emergency radiology512,
  11. Lotus Desbiolles, staff cardiologist512,
  12. Matthijs F L Meijs, resident in cardiology67,
  13. Maarten J Cramer, associate professor of cardiology6,
  14. Juhani Knuuti, professor, consultant, and director of centre8,
  15. Sami Kajander, consultant radiologist8,
  16. Jan Bogaert, adjunct chair of department of radiology9,
  17. Kaatje Goetschalckx, cardiologist9,
  18. Filippo Cademartiri, associate professor and head of cardiovascular imaging unit21011,
  19. Erica Maffei, staff radiologist1011,
  20. Chiara Martini, staff radiologist1011,
  21. Sara Seitun, radiologist10,
  22. Annachiara Aldrovandi, staff cardiologist10,
  23. Simon Wildermuth, professor of radiology and chairman of institute of radiology12,
  24. Björn Stinn, staff radiologist12,
  25. Jürgen Fornaro, staff radiologist12,
  26. Gudrun Feuchtner, associate professor of radiology13,
  27. Tobias De Zordo, resident in radiology13,
  28. Thomas Auer, resident in radiology13,
  29. Fabian Plank, research fellow13,
  30. Guy Friedrich, professor of medicine14,
  31. Francesca Pugliese, senior clinical lecturer and consultant15,
  32. Steffen E Petersen, reader in advanced cardiovascular imaging and honorary consultant cardiologist15,
  33. L Ceri Davies, reader in advanced cardiovascular imaging and honorary consultant cardiologist15,
  34. U Joseph Schoepf, professor of radiology, medicine, and pediatrics and director of cardiovascular imaging16,
  35. Garrett W Rowe, program coordinator16,
  36. Carlos A G van Mieghem, staff cardiologist17,
  37. Luc van Driessche, associate director of cardiology and consultant in interventional cardiology18,
  38. Valentin Sinitsyn, head of radiology department19,
  39. Deepa Gopalan, consultant cardiovascular radiologist20,
  40. Konstantin Nikolaou, professor of radiology and vice chair of department of clinical radiology21,
  41. Fabian Bamberg, fellow in radiology21,
  42. Ricardo C Cury, chairman of department of radiology and director of cardiac imaging22,
  43. Juan Battle, radiologist22,
  44. Pál Maurovich-Horvat, chairman of department of radiology and director of cardiac imaging23,
  45. Andrea Bartykowszki, resident in cardiology23,
  46. Bela Merkely, professor of cardiology and director of heart centre23,
  47. Dávid Becker, associate professor and deputy director heart centre23,
  48. Martin Hadamitzky, director of cardiac magnetic resonance imaging24,
  49. Jörg Hausleiter, assistant medical director24,
  50. Marc Dewey, chief consultant radiologist25,
  51. Elke Zimmermann, specialist in radiology25,
  52. Michael Laule, senior physician and deputy director of cardiac catheterisation laboratory26
  1. 1Department of Epidemiology, Erasmus University Medical Centre, Rotterdam, Netherlands
  2. 2Department of Radiology, Erasmus University Medical Centre
  3. 3Department of Public Health, Erasmus University Medical Centre
  4. 4Department of Cardiology, Erasmus University Medical Centre
  5. 5Institute of Diagnostic and Interventional Radiology, University Hospital Zurich, Zurich, Switzerland
  6. 6Department of Cardiology, University Medical Centre Utrecht, Utrecht, Netherlands
  7. 7Department of Radiology, University Medical Centre Utrecht
  8. 8Turku PET Centre, Turku University Hospital, Turku, Finland
  9. 9Department of Cardiovascular Diseases, University Hospital Leuven, Leuven, Belgium
  10. 10Department of Radiology and Cardiology, Azienda Ospedaliero-Universitaria, Parma, Italy
  11. 11Department of Radiology, Giovanni XXIII Clinic, Monastier, Treviso, Italy
  12. 12Institute of Radiology, Kantonsspital St Gallen, St Gallen, Switzerland
  13. 13Department of Radiology, Innsbruck Medical University, Innsbruck, Austria
  14. 14Department of Cardiology, Innsbruck Medical University
  15. 15Centre for Advanced Cardiovascular Imaging, Barts and The London NIHR Cardiovascular Biomedical Research Unit, Barts and the London School of Medicine and Dentistry, Barts and the London NHS Trust, London, UK
  16. 16Department of Radiology, Medical University of South Carolina, Charleston, SC, USA
  17. 17Department of Cardiology, Onze-Lieve-Vrouwziekenhuis Hospital Aalst, Aalst, Belgium
  18. 18Department of Cardiology, St Blasius Hospital Dendermonde, Belgium
  19. 19Department of Radiology, Federal Centre of Medicine and Rehabilitation, Moscow, Russia
  20. 20Department of Radiology, Papworth Hospital NHS Trust, Cambridge, UK
  21. 21Department of Clinical Radiology, University Hospitals Munich, Munich, Germany
  22. 22Department of Radiology, Baptist Hospital of Miami and Baptist Cardiac and Vascular Institute, Miami, FL, USA
  23. 23Heart Centre, Semmelweis University, Budapest, Hungary
  24. 24Department of Cardiology, German Heart Centre, Munich, Germany
  25. 25Department of Radiology, Charité, Medical School, Humboldt University, Berlin, Germany
  26. 26Department of Cardiology, Charité, Medical School
  27. 27Department of Health Policy and Management, Harvard School of Public Health, Harvard University, Boston, MA, USA
  1. Correspondence to: M G M Hunink, Departments of Epidemiology and Radiology, Erasmus University Medical Centre, PO Box 2040, 3000 CA Rotterdam, Netherlands m.hunink{at}erasmusmc.nl
  • Accepted 14 April 2012

Abstract

Objectives To develop prediction models that better estimate the pretest probability of coronary artery disease in low prevalence populations.

Design Retrospective pooled analysis of individual patient data.

Setting 18 hospitals in Europe and the United States.

Participants Patients with stable chest pain without evidence for previous coronary artery disease, if they were referred for computed tomography (CT) based coronary angiography or catheter based coronary angiography (indicated as low and high prevalence settings, respectively).

Main outcome measures Obstructive coronary artery disease (≥50% diameter stenosis in at least one vessel found on catheter based coronary angiography). Multiple imputation accounted for missing predictors and outcomes, exploiting strong correlation between the two angiography procedures. Predictive models included a basic model (age, sex, symptoms, and setting), clinical model (basic model factors and diabetes, hypertension, dyslipidaemia, and smoking), and extended model (clinical model factors and use of the CT based coronary calcium score). We assessed discrimination (c statistic), calibration, and continuous net reclassification improvement by cross validation for the four largest low prevalence datasets separately and the smaller remaining low prevalence datasets combined.

Results We included 5677 patients (3283 men, 2394 women), of whom 1634 had obstructive coronary artery disease found on catheter based coronary angiography. All potential predictors were significantly associated with the presence of disease in univariable and multivariable analyses. The clinical model improved the prediction, compared with the basic model (cross validated c statistic improvement from 0.77 to 0.79, net reclassification improvement 35%); the coronary calcium score in the extended model was a major predictor (0.79 to 0.88, 102%). Calibration for low prevalence datasets was satisfactory.

Conclusions Updated prediction models including age, sex, symptoms, and cardiovascular risk factors allow for accurate estimation of the pretest probability of coronary artery disease in low prevalence populations. Addition of coronary calcium scores to the prediction models improves the estimates.

Introduction

In the United States, about 10.2 million people have chest pain complaints each year,1 and more than 1.1 million diagnostic procedures of catheter based coronary angiography are performed on inpatients each year.2 In a recent report based on the national cardiovascular data registry of the American College of Cardiology,3 only 41% of patients undergoing elective procedures of catheter based coronary angiographies are diagnosed with obstructive coronary artery disease. The report’s authors concluded that better risk stratification was needed, underlined by decision analyses showing that the choice of further diagnostic investigation in patients with chest pain depends primarily on the pretest probability of coronary artery disease.4 5 6

The American College of Cardiology/American Heart Association,7 8 European Society of Cardiology,9 and United Kingdom10 currently recommend using the Diamond and Forrester model11 or the Duke clinical score12 13 to estimate the pretest probability of coronary artery disease in patients with chest pain. The Diamond and Forrester model tends to overestimate the probability of coronary artery disease (defined as ≥50% stenosis), and a revised version has recently been published.14 The Duke clinical score12 13 estimates the probability of coronary artery disease (≥75% stenosis) which, to our knowledge, has not been validated in populations outside the US. Although the American College of Cardiology/American Heart Association and European Society of Cardiology recommend exercise electrocardiography to select patients for further diagnostic investigation, UK guidelines recommend using the computed tomography (CT) based coronary calcium score in patients with a low to intermediate pretest probability (10-29%).

We perceived a need for an updated and stepwise approach to estimate the probability of coronary artery disease in patients with new onset of chest pain in a low prevalence population as clinical information and test results become available, in particular because implementation of the guidelines needs calculation of the pretest probability. Therefore, we aimed to estimate the probability of obstructive coronary artery disease on the basis of clinical presentation and cardiovascular risk factors, and to determine the incremental diagnostic value of exercise electrocardiography and the coronary calcium score.

Methods

Design overview

Researchers from Europe and the US formed a consortium. An existing database of at least 80 eligible patients was required for participation. Participation did not involve any financial incentives. All patients had to be enrolled in single centre studies, and local approval from the institutional review board for the original research objectives was required. The consortium is part of the European network for the assessment of imaging in medicine, which is an initiative of the European Institute of Biomedical Imaging Research.15 One of the network’s goals was to perform pooled analyses of existing prospectively collected data, improving power and increasing generalisability of results.

Participants

Patients were eligible for the analysis if they presented with stable chest pain and were referred for catheter based or CT based coronary angiography (≥64 slice). Patients were not eligible if they had acute coronary syndrome or unstable chest pain, had a history of myocardial infarction or previous revascularisation (percutaneous coronary intervention or coronary artery bypass graft surgery), or provided no informed consent. For the diagnosis of coronary artery disease, catheter based coronary angiography is regarded as the reference standard, which is an expensive and invasive procedure with a risk of complications. Non-invasive testing is generally recommended to select patients who might benefit from catheter based coronary angiography. CT based coronary angiography is a less invasive and less expensive test, with a high sensitivity and specificity for the detection of coronary artery disease on catheter based coronary angiography.16 17 18 A negative result from CT based coronary angiography virtually excludes the presence of obstructive coronary artery disease, whereas a positive result might need confirmation by the catheter based test.

Fourteen datasets consisted of consecutive patients enrolled in a prospective study for other research objectives. Four datasets consisted of patients retrospectively identified as eligible via electronic radiology reporting systems. Inclusion and exclusion criteria were evaluated by experienced doctors and missing information was obtained from patient records (web appendix table 1).19 20 21

Definitions

We collected data for age, sex, symptoms, cardiovascular risk factors, test results, and presence of coronary artery disease. Chest pain symptoms were classified as typical, atypical, or non-specific. Typical chest pain was defined as all of the following criteria: (1) substernal chest pain or discomfort that is (2) provoked by exertion or emotional stress and (3) relieved by rest or nitroglycerine (or both). We defined atypical chest pain as two of these criteria. If one or none of the criteria was present, symptoms were classified as non-specific.22

Definitions for hypertension, diabetes, dyslipidaemia, and smoking differed slightly across hospitals (web appendix table 1). Table 1 lists the most common definitions. We determined coronary calcium scores by the Agatston method23 and used log transformation to account for its skewed distribution.

Table 1

 Patient characteristics. Most commonly used risk factor definitions provided; characteristics data are no (%) of patients unless stated otherwise

View this table:

Outcomes

Primary outcome was obstructive coronary artery disease, defined as at least one vessel with at least 50% diameter stenosis found on catheter based coronary angiography. Since we combined existing databases from different hospitals, CT based and catheter based coronary angiographies were performed at each institution according to local protocols; we allowed both visual assessment and quantitative assessment for the interpretation of results for these procedures.

Statistical analysis

We assumed missing data occurred at random, depending on the clinical variables and the results of CT based coronary angiography, and performed multiple imputations using chained equations.24 Missing values were predicted on the basis of all other predictors considered, the results of CT based coronary angiography, as well as the outcome.25 26 We created 20 datasets with identical known information, but with differences in imputed values reflecting the uncertainty associated with imputations. In total, 667 (2%) clinical data items were imputed.

In our study, only a minority of patients underwent catheter based coronary angiography. An analysis restricted to patients who underwent catheter based coronary angiography could have been influenced by verification bias.27 Therefore, we imputed data for catheter based coronary angiography by using the CT based procedure as an auxiliary variable, in addition to all other predictors.28 Results for the two procedures correlate well together, especially for negative results of CT based coronary angiography.16 17 18 This strong correlation was confirmed in the 1609 patients who underwent both procedures (Pearson r=0.72). Since its data were used for imputation, the CT based procedure was not included as a predictor in the prediction models. Our approach was similar to using the results of CT based coronary angiography as the outcome variable when the catheter based procedure was not performed (which was explored in a sensitivity analysis). However, this approach is more sophisticated because it also takes into account other predictors and the uncertainty surrounding the imputed values. We imputed 3615 (64%) outcome values for catheter based coronary angiography. Multiple imputations were performed using Stata/SE 11 (StataCorp).

External validation of the Duke clinical score

To evaluate the performance of the Duke clinical score, we calculated the predicted probability based on published coefficients13 (that is, prediction of ≥75% stenosis). Since patients with evidence of previous coronary artery disease were excluded, we assumed all had a normal resting electrocardiogram. If resting findings were available and taken into account, any overestimation would increase further. We used a calibration plot to compare predicted probability with the observed proportions of severe disease (that is, ≥70% stenosis or ≥50% left main stenosis) in a calibration plot.

Development of new prediction models

We defined three prediction models: a basic model including age, sex, symptoms, and setting; a clinical model including age, sex, symptoms, setting, diabetes, hypertension, dyslipidaemia, smoking, and body mass index; and an extended model including all clinical variables and the coronary calcium score. Since all clinical variables are known to be associated with coronary artery disease,29 all predictors were entered simultaneously in a multivariable, random effects, logistic regression model. We included hospital as a random effect to account for clustering of patients within hospitals. Availability of data for exercise electrocardiography was limited, and web appendix table 5 explores the variable’s incremental predictive value. We omitted non-significant predictors with small effects (that is, odds ratio <1.01).

Setting variable

To account for differences in patient selection across datasets (based on referrals to catheter based coronary angiography v CT based coronary angiography), we created a dummy variable for setting. This variable was coded “0” (low prevalence setting) if a patient came from a database that was created by selecting patients who underwent CT based coronary angiography (of whom only a proportion underwent the catheter based procedure in addition to the CT based procedure), and coded as “1” (high prevalence setting) if a patient came from a database that was created by selecting patients who underwent catheter based coronary angiography (of whom a proportion also underwent the CT based procedure).

We intended to apply our prediction models to patients in low prevalence populations, for whom the best diagnostic management should be determined based on an estimated pretest probability.10 By contrast, all patients in the high prevalence setting had a clinical indication for catheter based coronary angiography, in whom estimating the pretest probability would not be relevant to them. However, because it would be inefficient to derive a prediction model in a low prevalence population only (since most patients will not undergo the reference standard), we also included databases with patients referred for catheter based coronary angiography. These data provided valuable information on the correlations between clinical presentation, risk factors, and the two angiography procedures, which was essential for reliable imputation of covariables and outcomes in the low prevalence populations. By including the setting variable, we could derive the model using all available data, and adjust for differences in patient selection. When applying the model for new patients with chest pain, the setting variable was set to zero.

Predictor effects might differ across the low and high prevalence settings, and we tested these differences by using interaction terms between setting and all other variables. We also tested interactions between symptoms and sex, symptoms and age, and symptoms and diabetes. Linear effects of age and the log transformed coronary calcium score were tested by including a restricted cubic spline function with three knots (df=2).30 31

We quantified diagnostic performance by calculating the area under the receiver operating characteristics curve (c statistic). Reclassification was assessed by use of the continuous net reclassification improvement (web appendix table 2).32 We regarded P<0.05 to be statistically significant. Analyses were performed using Stata/SE 11 (StataCorp).

Validation

We assessed the validity of the clinical model in a cross validation procedure. The four largest low prevalence databases with sufficient numbers for reliable validation,33 and the remaining low prevalence databases combined, were each in turn removed from the model development sample. We then validated each model using the database that was omitted during model development. We calculated the c statistic and validated the model according to the steps in the box (the web appendix provides more detail).26 30 31

Box: Cross validation procedure

Calibration-in-the-large

When assessing the validity of a prediction model, the first step is to check whether the average prediction approximates the average observed outcome. This concept is referred to as “calibration-in-the-large”. In this study, we compared the mean observed frequency of coronary artery disease in the validation data with the mean predicted probability. A negative value would imply that the mean observed proportion in the validation data is lower than the mean predicted probability. Conversely, a positive value would indicate that the mean observed proportion in the validation data was higher than the mean predicted probability. A statistically significant result indicates significant miscalibration, whereas a non-significant result supports validity of the prediction model

Logistic recalibration

The second step is to test whether the overall effect of the predictors in the model is valid for the validation data. The overall predictor effects in the validation data were re-estimated and compared with the reference overall predictor effects in the prediction model (reference is set to zero). A negative value would imply that the overall effects of the predictors in the validation data were lower than those in the prediction model. Conversely, a positive value would indicate that the overall effects of the predictors in the validation data were stronger than those in the prediction model. A significant result would indicate significant miscalibration of the predictor effects, whereas a non-significant result would indicate no difference in predictor effects, supporting the validity of the model

Re-estimation

The third step is to re-estimate the predictor effects in the validation data and calculate the difference (∂ coefficients) with the estimated predictor effects in the prediction model (web appendix).26 Non-significant differences (high P values) indicate no difference in the predictor effect, supporting the validity of the model

Results

Data collection and study population

We retrieved databases from 18 hospitals (table 1 and web appendix table 2). The study population included 5677 patients (3283 men, 2394 women; mean age 58 and 60 years, respectively). Nearly all patients (5190, 91%) underwent CT based coronary angiography, which revealed obstructive coronary artery disease in 1634 (31%). Of these 1634 patients, 1083 (66%) underwent catheter based coronary angiography, which showed positive results in 886 (82%). Of the 3556 patients without obstructive disease on CT based coronary angiography, 526 (15%) underwent catheter based coronary angiography, which showed negative results in 498 (95%). Overall, 2062 (36%) patients underwent catheter based coronary angiography, with 1176 (57%) diagnosed with obstructive coronary artery disease. Missing values occurred in four (0.1%) patients for age, six (0.1%) for symptoms, 126 (2.2%) for hypertension, 189 (3.3%) for diabetes, 187 (3.3%) for dyslipidaemia, 155 (2.7%) for smoking, 354 (6.2%) for body mass index, and 810 (14%) for coronary calcium score.

Of the 3556 patients who did not have obstructive coronary artery disease revealed by CT based coronary angiography, 3030 (85%) did not undergo the catheter based procedure. Results for catheter based coronary angiography were imputed for these patients, and were mostly negative (range 97-98.4% across the multiple imputations), which accords with the high negative predictive value of the CT based procedure. Of the 1634 patients who had obstructive disease revealed by the CT based procedure, 551 (34%) did not undergo subsequent catheter based coronary angiography. For these patients, results for the catheter based procedure were imputed, and were mostly positive (range 65-77% across imputations), which accords with a reduced positive predictive value of the CT based procedure.

External validation of the Duke clinical score

External validation of the Duke clinical score overestimated the probability of severe coronary artery disease, as observed in our dataset (fig 1 and web appendix table 3).

Figure1

Fig 1 Calibration plot of the Duke clinical score, in low prevalence datasets (n=4426). Distribution of predicted probabilities shown separately for patients with and without severe coronary artery disease. Triangles indicate observed proportions of severe disease, by tenths of predicted probability; 95% CI=confidence interval

Development of new prediction models

Table 2 summarises the results of the random effects analysis in logistic regression and the continuous net reclassification improvement (web appendix table 4 provides more detail). The prediction models are available as an online probability calculator (http://rcc.simpal.com/NpfpV5; web appendix fig A2). In the clinical model, all predictors except body mass index were significantly associated with obstructive coronary artery disease. The clinical model improved the prediction, compared with the basic model (cross validated c statistic improved from 0.77 to 0.79; table 2). Whereas an abnormal exercise electrocardiography had limited predictive value in the multivariable prediction model (web appendix table 5), the coronary calcium score was a major predictor, which increased the c statistic from 0.79 to 0.88 (table 2). Most predictor effects decreased after addition of the coronary calcium score; dyslipidaemia and smoking were no longer significant. We obtained similar results when using CT based coronary angiography as the outcome in patients who did not undergo catheter based coronary angiography.

Table 2

 Random effects logistic regression analysis* and cross validation, in the low prevalence setting

View this table:

Validation

Figure 2 and web appendix table 6 show the cross validation results for the clinical model. The c statistic ranged from 0.78 to 0.81. The continuous net reclassification improvement was a measure of the relative change in the observed proportion when the predicted probability changes (web appendix), which was most favourable (102%) for the extended model compared with the clinical model (table 2).

Figure2

Fig 2 Validity of clinical model using the four largest low prevalence datasets and the smaller remaining low prevalence databases combined. Distribution of predicted probabilities shown separately for patients with and without obstructive coronary artery disease. Triangles indicate observed proportion of disease, by tenths of the predicted probability; 95% CI=95% confidence interval

Assessment of calibration-in-the-large showed a significant difference between the mean observed outcome and the predicted probability (clinical model), for Azienda Ospedaliero Universitaria Parma, Rotterdam, and the combined low prevalence hospitals (fig 2). Logistic recalibration showed no significant differences between the overall hospital specific effects of the predictors compared with the overall effects of the predictors in the clinical model (fig 2). When re-estimated in specific datasets, the predictor effects were not significantly different from the predictor effects in the clinical model, except for the effect of typical chest pain for Azienda Ospedaliero Universitaria Parma. The results indicated that predictor effects were similar across datasets (web appendix table 6).

Discussion

Summary of key findings

With recently collected data and modern statistical methods, we developed a prediction model that performed well in estimating the probability of coronary artery disease. The need for an updated model was evident by our results showing that the Duke clinical score significantly overestimated the probability of coronary artery disease. Age, sex, symptoms, and coronary calcium score were strong predictors for disease. The clinical model predicted probabilities between 2% for a 50 year old woman with non-specific chest pain without any risk factors, and 91% for an 80 year old man with typical chest pain and multiple risk factors.

Previous publications

In 1979, Diamond and Forrester11 showed the importance of age, sex, and symptoms in the prediction of coronary artery disease. Despite its limitations, current guidelines still recommend the use of Diamond and Forrester’s model.7 8 As previously shown, the model tends to overestimate the probability of coronary artery disease, mainly in women,14 and does not take into account cardiovascular risk factors associated with the presence of the disease.

In 1993, Pryor and colleagues12 13 developed the Duke clinical score in a large cohort of patients who underwent catheter based coronary angiography. The model predicted the presence of at least 75% stenosis, on the basis of age, sex, symptoms, history of myocardial infarction, smoking, dyslipidaemia, diabetes, and resting electrocardiography findings. In our study, we showed that the Duke clinical score also overestimated the probability of coronary artery disease.

Limitations of the study

The study population was derived from existing databases, some of which were designed for other research objectives (such as to investigate the diagnostic accuracy of CT based coronary angiography for coronary artery disease). In some studies, all patients underwent the reference standard test, whereas other studies selected patients for catheter based coronary angiography on the basis of results from the CT based procedure. When evaluating the diagnostic performance of CT based coronary angiography, selection on the basis of the test results could lead to verification bias or selective referral bias, which could affect the sensitivity and specificity estimates of such a test. However, in the current study, we did not assess the diagnostic value of the CT based procedure, thus making bias less likely. We also considered patients who underwent the CT based procedure only and not the catheter based procedure. We selected patients who underwent either angiography, or both procedures. However, not all patients presenting with chest pain will be referred for one procedure or the other, which limits the generalisability of our results.

To also consider patients who did not undergo catheter based coronary angiography, we multiply imputed these results on the basis of results from the CT based procedure and all other covariables. Although data for the catheter based procedure were imputed for a large proportion of patients, we believe that the high sensitivity and specificity of the CT based procedure justifies the imputation; we also did not consider the CT based procedure as an explanatory variable in the prediction models. However, the CT based procedure could overestimate the severity of disease, which in turn could have overestimated the imputed severity of disease from the catheter based procedure. If bias was present in our models, it would tend to advocate further diagnostic investigation rather than missing the diagnosis.

We combined existing data from several different hospitals. Since the current analysis was not the main purpose of the data collection, the selection of patients, availability of data, and predictor definitions differed across hospitals (web appendix table 1). Furthermore, some hospitals used quantitative coronary angiography to determine the degree of stenosis, whereas others used visual assessment. Overall, heterogeneity due to differences between protocols, level of physician experience, and guideline adherence across hospitals could have influenced our results. Despite these limitations, the models presented had generally good discrimination (via the c statistic) and calibration.

Because we intended to use our model in low prevalence populations, cross validation was performed using only the low prevalence datasets. The cross validation results in the data from Parma, Rotterdam, and the smaller hospitals combined, were less favourable in terms of calibration-in-the-large, possibly explained by heterogeneity. However, in general, calibration assessed graphically (fig 2) could be considered satisfactory, suggesting that the model is generalisable to other settings. Further external validation of our model in other populations is still needed.

Our study focused on the prediction of obstructive coronary artery disease according to age, sex, type of chest pain, cardiovascular risk factors, setting, and coronary calcium score. Unfortunately, no data were available to assess the predictive value of findings based on other imaging tests (such as nuclear perfusion imaging, perfusion magnetic resonance imaging, and echocardiography). Furthermore, we were unable to show incremental predictive value of exercise electrocardiography, which may be explained by the limited availability of data.

Finally, our analysis focused on the diagnostic prediction of the presence of coronary artery disease, defined as at least 50% diameter stenosis in at least one vessel, shown on catheter based coronary angiography. Predicting severe disease (for example, ≥70% stenosis in the left anterior descending coronary artery, three vessel disease, or left main coronary artery disease) would be helpful as a tool to select patients for revascularisation. However, the main purpose of the current analysis was the development of prediction models to help physicians select patients who would benefit from further testing.

Future directions

Other non-invasive imaging tests for the evaluation of patients with chest pain should also be considered as predictors for the presence of coronary artery disease. Furthermore, cost effectiveness analyses should be performed to establish appropriate thresholds for diagnostic testing, taking into account the long term benefits and harms.4 34

Clinical implications

We showed that the Duke clinical score overestimates the likelihood of coronary artery disease, and we believe that our model improves the estimate of the pretest probability. By contrast with the Duke clinical score, we developed and validated our model using data from different hospitals, settings, countries, and included hypertension as a predictor. Finally, our model does not need inclusion of resting electrocardiography findings, which could be convenient in primary practice.

A refined estimate of the probability of coronary artery disease allows doctors to make better decisions as to which diagnostic test is best in a particular patient, according to NICE guidelines, and to decide on further management based on the results of such tests. Our stepwise models can be used to evaluate the added value of performing coronary calcium score either before performing the test or after obtaining the calcium score. Our analysis shows that the coronary calcium score significantly improves the estimate of the probability of coronary artery disease, suggesting that the score should be considered for patients with chest pain. Use of the coronary calcium score is not routinely recommended by the guidelines from the American College of Cardiology/American Heart Association35 or the European Society of Cardiology,9 whereas the UK guidelines recommend use of the score if the pretest probability is 10-29%.10 However, triage strategies using the coronary calcium score have been proposed35 and are implemented in several centres. In this context, our prediction calculator could be used both to determine whether using the coronary calcium score is clinically useful (by checking whether the score alters the probability of coronary artery disease such that clinical management would change) and to determine the revised probability of coronary artery disease, based on the score’s result.

Our findings also suggest that the diagnostic value of exercise electrocardiography is limited (web appendix), which accords with its low sensitivity and specificity for detecting coronary artery disease,36 and with explicit recommendations in the UK not to use the test to diagnose or exclude disease in patients with chest pain.10 However, many physicians argue that the prognostic information obtained by exercise electrocardiography remains important for clinical practice. Alternatively, coronary calcium scores provide both diagnostic37 38 39 and prognostic information.40 41 42 43

Prediction tools are useful only when they are easily accessible at the point of care, which is why we designed an online calculator (web appendix fig A2). The calculator could be implemented in electronic patient records, electronic order entry systems, or smartphone or tablet applications. Overall, prediction models that include age, sex, symptoms, and risk factors allow for accurate estimation of the probability of coronary artery disease in low prevalence populations. The addition of using the coronary calcium score improved the prediction of the disease. Implementation of these models could improve clinical outcomes, but would need further evaluation.

What is already known on this topic

  • Current guidelines recommend use of the Diamond and Forrester model or Duke clinical score to estimate the pretest probability of coronary artery disease in patients presenting with stable chest pain

  • The pretest probability of disease helps physicians identify which patients are most likely to benefit from further testing

What this study adds

  • The Duke clinical score overestimates the probability of coronary artery disease

  • An updated prediction model for low prevalence populations was developed and validated using individual patient data from 18 different hospitals

  • Use of the CT based coronary calcium score in the updated model improved the predicted probability of disease

Notes

Cite this as: BMJ 2012;344:e3485

Footnotes

  • We thank C Greg Hagerty and Michael W Kattan from the Cleveland Clinic Lerner Research Institute (Cleveland, OH, USA) for developing, customising, and improving the Cleveland clinic risk calculator constructor (http://rcc.simpal.com/).

  • Funding: This research was supported by a healthcare efficiency grant from the Erasmus University Medical Centre. The authors’ work was independent of the funding sources. The funding organisations had no involvement in design and conduct of the study; collection, management, analysis and interpretation of the data; preparation, review, or approval of the manuscript; and in the decision to publish the manuscript.

  • Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: support from the Erasmus University Medical Centre; FP, SEP, and LCD’s work contributes to the translational research portfolio of Barts and the London Cardiovascular Biomedical Research Unit, which is supported and funded by the National Institute for Health Research; UJS is a consultant for Bayer and Siemens and provides research support to Bayer, Bracco, General Electric, and Siemens; MD holds research grants (European Regional Development Fund, German Heart Foundation/German Foundation of Heart Research, GE Healthcare (Amersham), Bracco, Guerbet, and Toshiba Medical Systems), receives speaker fees (Toshiba Medical Systems, Guerbet, and Bayer-Schering), runs cardiac CT workshops at Charité (www.ct-kurs.de), has written the book Coronary CT Angiography (Springer, 2008), and is involved in institutional research collaborations (Siemens Medical Solutions, Philips Medical Systems, and Toshiba Medical Systems); FC is a consultant for Servier, received speaker fees (Bracco Imaging), and holds a research grant with HE Healthcare; EM is a consultant for Servier and holds a research grants with HE Healthcare; FB has received speakers fees from Siemens Healthcare; all other authors have no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years and no other relationships or activities that could appear to have influenced the submitted work.

  • Ethical approval: Not required.

  • Contributors: MGMH, EWS, and TSSG participated in the study concept and design, analysis and interpretation of data, and drafting of the manuscript. All authors apart from MGMH, EWS, and TSSG were involved in the acquisition of data. All authors contributed to the critical revision of the manuscript for important intellectual content and final approval of the published version. MGMH, the principal investigator and guarantor, had full access to all the data in the study and takes responsibility for the integrity of the data and accuracy of the data analysis, and was responsible for manuscript preparation and decision to submit the manuscript for publication.

  • Data sharing: No additional data available.

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

References

View Abstract