Risk models and scores for type 2 diabetes: systematic review
BMJ 2011; 343 doi: https://doi.org/10.1136/bmj.d7163 (Published 28 November 2011) Cite this as: BMJ 2011;343:d7163- Douglas Noble, lecturer1,
- Rohini Mathur, research fellow1,
- Tom Dent, consultant2,
- Catherine Meads, senior lecturer1,
- Trisha Greenhalgh, professor1
- 1Centre for Primary Care and Public Health, Barts and the London School of Medicine and Dentistry, London E1 2AT, UK
- 2PHG Foundation, Cambridge, UK
- Correspondence to: D Noble d.noble{at}qmul.ac.uk
- Accepted 5 October 2011
Abstract
Objective To evaluate current risk models and scores for type 2 diabetes and inform selection and implementation of these in practice.
Design Systematic review using standard (quantitative) and realist (mainly qualitative) methodology.
Inclusion criteria Papers in any language describing the development or external validation, or both, of models and scores to predict the risk of an adult developing type 2 diabetes.
Data sources Medline, PreMedline, Embase, and Cochrane databases were searched. Included studies were citation tracked in Google Scholar to identify follow-on studies of usability or impact.
Data extraction Data were extracted on statistical properties of models, details of internal or external validation, and use of risk scores beyond the studies that developed them. Quantitative data were tabulated to compare model components and statistical properties. Qualitative data were analysed thematically to identify mechanisms by which use of the risk model or score might improve patient outcomes.
Results 8864 titles were scanned, 115 full text papers considered, and 43 papers included in the final sample. These described the prospective development or validation, or both, of 145 risk prediction models and scores, 94 of which were studied in detail here. They had been tested on 6.88 million participants followed for up to 28 years. Heterogeneity of primary studies precluded meta-analysis. Some but not all risk models or scores had robust statistical properties (for example, good discrimination and calibration) and had been externally validated on a different population. Genetic markers added nothing to models over clinical and sociodemographic factors. Most authors described their score as “simple” or “easily implemented,” although few were specific about the intended users and under what circumstances. Ten mechanisms were identified by which measuring diabetes risk might improve outcomes. Follow-on studies that applied a risk score as part of an intervention aimed at reducing actual risk in people were sparse.
Conclusion Much work has been done to develop diabetes risk models and scores, but most are rarely used because they require tests not routinely available or they were developed without a specific user or clear use in mind. Encouragingly, recent research has begun to tackle usability and the impact of diabetes risk scores. Two promising areas for further research are interventions that prompt lay people to check their own diabetes risk and use of risk scores on population datasets to identify high risk “hotspots” for targeted public health interventions.
Introduction
The prevalence of diabetes is rising rapidly throughout the world.1 By 2010 its prevalence in the adult populations of the United Kingdom, the United States, mainland China, and the United Arab Emirates had exceeded 7%,2 11%,3 15%,4 and 17%,5 respectively. Americans born in 2000 or later have a lifetime risk of more than one in three of developing diabetes.6 Type 2 diabetes (which accounts for over 95% of diabetes worldwide) results from a complex gene-environment interaction for which several risk factors, such as age, sex, ethnicity, family history, obesity, and hypertension, are well documented. The precise interaction of these and other risk factors with one another is, however, a complex process that varies both within and across populations.7 8 9 10 11 Epidemiologists and statisticians are striving to produce weighted models that can be presented as scores to reflect this complexity but which at the same time are perceived as sufficiently simple, plausible, affordable, and widely implementable in clinical practice.12 13
Cohort studies have shown that early detection of established diabetes improves outcome, although the evidence base for screening the entire population is weak.14 15 The proportion of cases of incident type 2 diabetes in people with impaired glucose tolerance or impaired fasting glucose levels was reduced in landmark trials from China,16 Finland,17 and the United States18 by up to 33%, 50%, and 58%, respectively, through lifestyle changes (increased exercise, weight loss) or pharmacotherapy, or both, although changes may be more modest in a non-trial population. Some have argued that because combining known risk factors predicts incident diabetes at least as effectively as impaired glucose metabolism, a diabetes risk score may be a better and more practical means of identifying people for preventive interventions than either a glucose tolerance test or a fasting blood glucose level.19 Others favour targeting the assessment of diabetes risk in those with established impaired glucose metabolism on the basis that interventions in this group are particularly effective.20
Risk models and scores first emerged for cardiovascular disease, and these are widely used in clinical and public health practice. In the United Kingdom, for example, all electronic patient record systems in general practice offer the facility to calculate the Framingham score, a patient’s risk of a cardiovascular event within 10 years. This risk score features in many guidelines and decision pathways (such as the cut-off for statin therapy21), and general practitioners receive financial rewards for calculating it.22 In contrast, although numerous models and scores have been developed for diabetes risk, we found limited evidence for use of these as part of a formal health policy, guideline, or incentive scheme for practitioners in any country (one Australian scheme incentivises general practitioners’ measurement of the risk of diabetes in adults aged 40-4923). This is perhaps surprising, given that morbidity and mortality from cardiovascular disease has been decreasing in many countries since the 1970s,24 whereas those from diabetes continue to increase.3
A diabetes risk score is an example of a prognostic model.25 Such scores should ideally be developed by taking a large, age defined population cohort of people without diabetes, measuring baseline risk factors, and following the cohort for a sufficiently long time to see who develops diabetes.26 Although prospective longitudinal designs in specially assembled cohorts are expensive, difficult, and time consuming to execute, cross sectional designs in which risk factors are measured in a population including people both with and without diabetes are methodologically inferior. They use prevalence as a proxy for incidence and conflate characteristics of people with diabetes with risk factors in those without diabetes, and thus are incapable of showing that a putative risk factor predated the development of diabetes. In practice, researchers tend to take one of two approaches: they either study a cohort of people without diabetes, which was assembled some years previously with relevant baseline metrics for some other purpose (for example, the British Regional Heart Study27), or they analyse routinely available data, such as electronic patient records.8 Both approaches are potentially susceptible to bias.
Some diabetes risk scores are intended to be self administered using questions such as “have you ever been told you have high blood pressure?” Scores that rely entirely on such questions may be hosted on the internet (see for example www.diabetes.org.uk/riskscore). Some researchers have used self completion postal questionnaires as the first part of a stepwise detection programme.28 To the extent that these instruments are valid, they can identify two types of people: those who already have diabetes whether or not they know it (hence the questionnaire may serve as a self administered screening tool for undiagnosed diabetes) and those at high risk of developing diabetes—that is, it may also serve as a prediction tool for future diabetes. Prevalence rates for diabetes derived from self assessment studies thus cannot be compared directly with the rate of incident diabetes in a prospective longitudinal sample from which those testing positive for diabetes at baseline have been excluded.
A good risk score is usually defined as one that accurately estimates individuals’ risk—that is, predictions based on the score closely match what is observed (calibration); the score distinguishes reliably between high risk people, who are likely to go on to develop the condition, and low risk people, who are less likely to develop the condition (discrimination); and it performs well in new populations (generalisability). Validating a risk model or score means testing its calibration and discrimination either internally (by splitting the original sample, developing the score on one part and testing it on another), temporally (re-running the score on the same or a similar sample after a time period), or, preferably, externally (running the score on a new population with similar but not identical characteristics from the one on which it was developed).26 29 Caution is needed when extrapolating a risk model or score developed in one population or setting to a different one—for example, secondary to primary care, adults to children, or one ethnic group to another.30
Risk scores and other prognostic models should be subject to “impact studies”—that is, studies of the extent to which the score is actually used and leads to improved outcomes. Although most authors emphasise quantitative evaluation of impact such as through cluster randomised controlled trials,30 much might also be learnt from qualitative studies of the process of using the score, either alone or as an adjunct to experimental trials. One such methodology is realist evaluation, which considers the interplay between context, mechanism (how the intervention is perceived and taken up by practitioners), and outcome.31 In practice, however, neither quantitative nor qualitative studies of impact are common in the assessment of risk scores.30
We sought to identify, classify, and evaluate risk models and scores for diabetes and inform their selection and implementation in practice. We wanted to determine the key statistical properties of published scores for predicting type 2 diabetes in adults and how they perform in practice. Hence we were particularly interested in highlighting those characteristics of a risk score that would make it fit for purpose in different situations and settings. To that end we reviewed the literature on development, validation, and use of such scores, using both quantitative data on demographics of populations and statistical properties of models and qualitative data on how risk scores were perceived and used by practitioners, policy makers, and others in a range of contexts and systems.
Methods
Theoretical and methodological approach
We followed standard methodology for systematic reviews, summarised in guidance from a previous study and the York Centre for Reviews and Dissemination.32 33 The process was later extended by drawing on the principles of realist review, an established form of systematic literature review that uses mainly qualitative methods to produce insights into the interaction between context, mechanism, and outcome, hence explaining instances of both success and failure.34 Our team is leading an international collaborative study, the Realist and Meta-narrative Evidence Synthesis: Evolving Standards (RAMESES) to develop methodological guidance and publication standards for realist review.35
Search strategy
We identified all peer reviewed cohort studies in adults over age 18 that had derived or validated, or both, a statistically weighted risk model for type 2 diabetes in a population not preselected for known risk factors or disease, and which could be applied to another population. Studies were included that had developed a new risk model based on risk factors and that used regression techniques to weight risk factors appropriately, or validated an existing model on a new population, or did both. Exclusion criteria were cross sectional designs, studies that had not finished recruiting, studies on populations preselected for risk factors or disease, studies that did not link multiple risk factors to form a scoring system or weighted model, screening or early detection studies, genetic studies, case series, studies on under 18s, animal studies, and studies that applied a known risk model or score to a population but did not evaluate its statistical potential.
In January 2011 we undertook a scoping search, beginning with sources known to the research team and those recommended by colleagues. We used the 29 papers from this search to develop the definitive protocol, including search terms and inclusion and exclusion criteria. In February 2011 a specialist librarian designed a search strategy (see web extra) with assistance from the research team. Key words were predict, screen, risk, score, [type two] diabetes, model, regression, risk assessment, risk factor, calculator, analysis, sensitivity and specificity, ROC and odds ratio. Both MESH terms and text words were used. Titles and abstracts were searched in Medline, PreMedline, Embase, and relevant databases in the Cochrane Library from inception to February 2011, with no language restrictions.
Search results from the different databases were combined in an endnote file and duplicates removed electronically and manually. In February and March 2011 two researchers independently scanned titles and abstracts and flagged potentially relevant papers for full text analysis.
Two researchers independently read the interim dataset of full text papers and reduced this to a final dataset of studies, resolving disagreements by discussion. Bilingual academic colleagues translated non-English papers and extracted data in collaboration with one of the research team. To identify recently published papers two researchers independently citation tracked the final dataset of studies in Google Scholar. Reference lists of the final dataset and other key references were also scanned.
Quantitative data extraction and analysis
Properties of included studies were tabulated on an Excel spreadsheet. A second researcher independently double checked the extraction of primary data from every study. Discrepancies were resolved by discussion. Where studies trialled multiple models with minimal difference in the number of risk factors, a judgment was made to extract data from the authors’ preferred models or (if no preferences were stated in the paper) the ones judged by two researchers to be the most complete in presentation of data or statistical robustness. Data extraction covered characteristics of the population (age, sex, ethnicity, etc), size and duration of study, completeness of follow-up, method of diagnosing diabetes, details of internal or external validation, or both, and the components and metrics used by authors of these studies to express the properties of the score, especially their calibration and discrimination—for example, observed to predicted ratios, sensitivity and specificity, area under the receiver operating characteristic curve. We aimed to use statistical meta-analysis where appropriate and presented heterogeneous data in disaggregated form.
Qualitative data extraction and analysis
For the realist component of the review we extracted data and entered these on a spreadsheet under seven headings (box 1).
Box 1: Categories for data entry
Intended users
Authors’ assumptions (if any) about who would use the risk score, on which subgroups or populations
Proposed action based on the score result
Authors’ assumptions (if any) on what would be offered to people who score above the designated cut-off for high risk
Mechanism
Authors’ hypothesised (or implied) mechanism by which use of the score might improve outcomes for patients
Descriptor
Authors’ adjectives to describe their risk model or score
Relative advantage
Authors’ claims for how and in what circumstances their model or score outperforms previous ones
Concerns
Authors’ stated concerns about their model or score
Real world use, including citation tracking
Actual data in this paper or papers citing it on use of the score in the real world
One researcher extracted these data from our final sample of papers and another checked a one third sample of these. Our research team discussed context-mechanism-outcome interactions hypothesised or implied by authors and reread the full sample of papers with all emerging mechanisms in mind to explore these further.
Impact analysis
We assessed the impact of each risk score in our final sample using three criteria: any description in the paper of use of the score beyond the population for whom it was developed and validated; number of citations of the paper in Google Scholar and number of these that described use of the score in an impact study; and critical appraisal of any impact studies identified on this citation track. In this phase we were guided by the question: what is the evidence that this risk score has been used in an intervention which improved (or sought to improve) outcomes for individuals at high risk of diabetes?
Prioritising papers for reporting
Given the large number of papers, statistical models, and risk scores in our final sample, we decided for clarity to highlight a small number of scores that might be useful to practising clinicians, public health specialists, or lay people. Adapting previous quality criteria for risk scores,26 we favoured those that had external validation by a separate research team on a different population (generalisability), statistically significant calibration, a discrimination greater than 0.70, and 10 or fewer components (usability).
Results
Figure 1⇓ shows the flow of studies through the review. One hundred and fifteen papers were analysed in detail to produce a final sample of 43. Of these 43 papers, 18 described the development of one or more risk models or scores,8 27 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 17 described external validation of one or more models or scores on new populations,9 10 19 52 53 54 55 56 57 58 59 60 61 62 63 64 65 and eight did both.7 66 67 68 69 70 71 72 In all, the 43 papers described 145 risk models and scores, of which 94 were selected for extraction of full data (the other 51 were minimally different, were not the authors’ preferred model, or lacked detail or statistical robustness). Of the final sample of 94 risk models, 55 were derivations of risk models on a base population and 39 were external validations (of 14 different models) on new populations. Studies were published between 1993 and 2011, but most appeared in 2008-11 (fig 2⇓). Indeed, even given that weaker cross sectional designs had been excluded, the findings suggest that new risk models and scores for diabetes are currently being published at a rate of about one every three weeks.
Table 1⇓ gives full details of the studies in the sample, including the origin of the study, setting, population, methodological approach, duration, and how diabetes was diagnosed. The studies were highly heterogeneous. Models were developed and validated in 17 countries representing six continents (30 in Europe, 25 in North America, 21 in Asia, 8 in Australasia, 8 in the Middle East, 1 in South America, and 1 in Africa).
Comparisons across studies were problematic owing to heterogeneity of data and highly variable methodology, presentation techniques, and missing data. Cohorts ranged in size from 399 to 2.54 million. The same data and participants were often included in several different models in the same paper. Ten research populations were used more than once in different papers.9 10 27 37 41 42 44 46 47 48 49 51 52 53 54 55 56 63 64 65 66 70 71 In total, risk models were tested on 6.88 million participants, although this figure includes duplicate tests on the same dataset. Participants aged 18 to 98 were studied for periods ranging from 3.15 to 28 years. Completeness of follow-up ranged from 54% to 99% and incidence of diabetes across the time periods studied ranged from 1.3% to 20.9%.
None of the models in the sample was developed on a cohort recruited prospectively for the express purpose of devising it. Rather, all authors used the more pragmatic approach of retrospectively studying a research dataset that had been assembled some years previously for a different purpose. Forty two studies excluded known diabetes in the inception cohort. Diagnosis of diabetes in a cohort at inception and completion of the study was done in different ways, including self report, patient questionnaires, clinician diagnosis, electronic code, codes from the International Classification of Diseases, disease or drug registers, diabetes drugs, dietary treatment, fasting plasma glucose levels, oral glucose tolerance test, and measurement of haemoglobin A1c. In some studies the method was not stated. Half the studies used different diagnostic tests at inception and completion of the study.
One third of the papers focused almost exclusively on the statistical properties of the models. Many of the remainder had a clinician (diabetologist or general practitioner) as coauthor and included an (often short and speculative) discussion on how the findings might be applied in clinical practice. Three described their score as a “clinical prediction rule.”45 51 59
Quantitative findings
Table 2⇓ gives details of the components of the 94 risk models included in the final sample and their statistical properties—including (where reported) their discrimination, calibration, sensitivity, specificity, positive and negative predictive value, and area under the receiver operating characteristic curve. Many papers offered additional sophisticated statistical analysis, although there was no consistency in the approach used or statistical tests. Heterogeneity of data (especially demographic and ethnic diversity of validation cohorts and different score components) in the primary studies precluded formal meta-analysis.
All 94 models presented a combination of risk factors as significant in the final model, and different models weighted different components differently. The number of components in a single risk score varied from 3 to 14 (n=84, mean 7.8, SD 2.6). The seven risk scores that were classified as having high potential for use in practice offered broadly similar components and had similar discriminatory properties (area under receiver operating characteristic curve 0.74-0.85, table 4). Overall, the areas under the receiver operating characteristic curve ranged from 0.60 to 0.91. Certain components used in some models (for example, biomarkers) are rarely available in some pathology laboratories and potentially too expensive for routine use. Some models that exhibited good calibration and discrimination on the internal validation cohort performed much less well when tested on an external cohort,62 67 suggesting that the initial model may have been over-fitted by inclusion of too many variables that had only minor contributions to the total risk.73 Although this study did not seek out genetic components, those studies that had included genetic markers alongside sociodemographic and clinical data all found that the genetic markers added little or nothing to the overall model.9 10 36 50
Reporting of statistical data in some studies was incomplete—for example, only 40 of the 94 models quantified any form of calibration statistic. Forty three presented sensitivity and specificity, 27 justified the rationale for cut-off points, 22 presented a positive predictive value, 19 presented a negative predictive value, and 26 made some attempt to indicate the percentage of the population that would need clinical follow-up or testing if they scored as “high risk.” Some models performed poorly—for example, there was a substantial gap between expected and observed numbers of participants who developed diabetes over the follow-up period. The false positive and false negative rates in many risk scores raised questions about their utility in clinical practice (for example, positive predictive values ranged from 5% to 42%, negative predictive values from 88% to 99%). However, some scores were designed as non-invasive preliminary instruments, with a recommended second phase involving a blood test.7 43 52 53 55 58 65
Risk models and scores tended to “morph” when they were externally validated because research teams dropped components from the original (for example, if data on these were not available), added additional components (for example, to compensate for missing categories), or modified what counted in a particular category (for example, changing how ethnicity was classified); in some cases these modifications were not clarified. A key dimension of implementation is appropriate adaptation to a new context. It was considered that this did not negate the external validation.
Qualitative findings
Table 3⇓ provides the qualitative findings from the risk scores. Of the 43 papers in the full sample, three did not recommend use of the model tested because the authors believed it had no advantage over existing ones.50 56 60 Authors of the other 40 papers considered that at least one of their scores should be adopted and used, and to justify this made various claims. The commonest adjective used by authors to describe their score was “simple” (26 of 43); others mentioned “low cost,” “easily implemented,” “feasible,” and “convenient.”
Sixteen of the 43 studies that recommended use of a particular risk model or score did not designate an intended user for it. Some authors assigned agency to a risk score—that is, they stated, perhaps inadvertently, that the score itself had the potential to prevent diabetes, change behaviour, or reduce health inequalities. Although most authors did state an intended target group, this was usually given in vague terms, such as “the general population” or “individuals who are likely to develop diabetes.” Eleven of the 43 papers gave a clear statement of what intervention might be offered, by whom, to people who scored above the cut-off for high risk; the other papers made no comment on this or used vague terms such as “preventive measures,” without specifying by whom these would be delivered.
In all, authors of the papers in the full sample either explicitly identified or appeared to presume 10 mechanisms (box 2) by which, singly or in combination, use of the diabetes risk score might lead to improved patient outcomes (see table 3).
Box 2: 10 suggested mechanisms by which diabetes risk scores could help improve patient outcomes
Clinical
Direct impact—clinicians will pick up high risk patients during consultations and offer advice that leads to change in patients’ behaviour and lifestyle
Indirect impact—routine use of the score increases clinicians’ awareness of risk for diabetes and motivation to manage it
Self assessment
Direct impact—people are alerted by assessing their own risk (for example, using an online tool), directly leading to change in lifestyle
Indirect impact—people, having assessed their own risk, are prompted to consult a clinician to seek further tests or advice on prevention
Technological
Individual impact—a risk model programmed into the electronic patient record generates a point of care prompt in the clinical encounter
Population impact—a risk model programmed into the electronic patient record generates aggregated data on risk groups, which will inform a public health intervention
Public health
Planners and commissioners use patterns of risk to direct resources into preventive healthcare for certain subgroups
Administrative
An administrator or healthcare assistant collects data on risk and enters these onto the patients’ records, which subsequently triggers the technological, clinical, or public health mechanisms
Research into practice
Use of the risk score leads to improved understanding of risk for diabetes or its management by academics, leading indirectly to changes in clinical practice and hence to benefits for patients
Future research
Use of the risk score identifies focused subpopulations for further research (with the possibility of benefit to patients in later years)
Risk models and scores had been developed in a range of health systems. Differences in components could be explained partly in terms of their intended context of use. For example, the QDScore, intended for use by general practitioners, was developed using a database of electronic records of a nationally representative sample of the UK general practice population comprising 2.5 million people. The QDScore is composed entirely of data items that are routinely recorded on general practice electronic records (including self assigned ethnicity, a deprivation score derived from the patient’s postcode, and clinical and laboratory values).8 Another score, also intended to be derived from electronic records but in a US health maintenance organisation (covering people of working age who are in work), has similar components to the QDScore except that ethnicity and socioeconomic deprivation are not included. In contrast, the FINDRISC score was developed as a population screening tool intended for use directly by lay people; it consists of questions on sociodemographic factors and personal history along with waist circumference but does not include clinical or laboratory data; high scorers are prompted to seek further advice from a clinician.52 Such a score makes sense in many parts of Finland and also in the Netherlands where health and information literacy rates are high, but would be less fit for purpose in a setting where these were low.
Prioritising scores for practising clinicians
Table 4⇓ summarises the properties of seven validated diabetes risk scores which we judged to be the most promising for use in clinical or public health practice. The judgments on which this selection was based were pragmatic; other scores not listed in table 4 (also see tables 1 and 2) will prove more fit for purpose in certain situations and settings. One score that has not yet been externally validated was included in table 4 because it is the only score already being incentivised in a national diabetes prevention policy.23
Studies of impact of risk scores on patient outcomes
None of the 43 papers that validated one or more risk scores described the actual use of that score in an intervention phase. Furthermore, although these papers had been cited by a total of 1883 (range 0-343, median 12) subsequent papers, only nine of those 1883 papers (table 5⇓) described application and use of the risk score as part of an impact study aimed at changing patient outcomes. These covered seven studies, of which (to date) three have reported definitive results. All three reported positive changes in individual risk factors, but surprisingly none recalculated participants’ risk scores after the intervention period to see if they had changed. While one report on the ongoing FIN-D2D study suggests that incident diabetes has been reduced in “real world” (non-trial) participants who were picked up using a diabetes risk score and offered a package of preventive care,74 this is a preliminary and indirect finding based on drug reimbursement claims, and no actual data are given in the paper. With that exception, no published impact study on a diabetes risk score has yet shown a reduction in incident diabetes.
Discussion
Numerous diabetes risk scores now exist based on readily available data and provide a good but not perfect estimate of the chance of an adult developing diabetes in the medium term future. A few research teams have undertaken exemplary development and validation of a robust model, reported its statistical properties thoroughly, and followed through with studies of impact in the real world.
Limitations of included studies
We excluded less robust designs (especially cross sectional studies). Nevertheless, included studies were not entirely free from bias and confounding. This is because the “pragmatic” use of a previously assembled database or cohort brings an inherent selection bias (for example, the British Regional Heart Study cohort was selected to meet the inclusion criteria for age and comorbidity defined by its original research team and oriented to research questions around cardiovascular disease; the population for the QDScore is drawn from general practice records and hence excludes those not registered with a general practitioner).
Most papers in our sample had one or more additional limitations. They reported models or scores that required collection of data not routinely available in the relevant health system; omitted key statistical properties such as calibration and positive and negative predictive values that would allow a clinician or public health commissioner to judge the practical value of the score; or omitted to consider who would use the score, on whom, and in what circumstances. We identified a mismatch between the common assumption of authors who develop a risk model (that their “simple” model can now be taken up and used) and the actual uptake and use of such models (which seems to happen very rarely). However, there has recently been an encouraging—if limited—shift in emphasis from the exclusive pursuit of statistical elegance (for example, maximising area under the receiver operating curve) to undertaking applied research on the practicalities and outcomes of using diabetes risk scores in real world prevention programmes.
Strengths and limitations of the review
The strengths of this review are our use of mixed methodology, orientation to patient relevant outcomes, extraction and double checking of data by five researchers, and inclusion of a citation track to identify recently published studies and studies of impact. We applied both standard systematic review methods (to undertake a systematic and comprehensive search, translate all non-English texts, and extract and analyse quantitative data) and realist methods (to consider the relation between the components of the risk score, the context in which it was intended to be used, and the mechanism by which it might improve outcomes for patients).
The main limitation of this review is that data techniques and presentation in the primary studies varied so much that it was problematic to determine reasonable numerators and denominators for many of the calculations. This required us to make pragmatic decisions to collate and present data as fairly and robustly as possible while also seeking to make sense of the vast array of available risk scores to the general medical reader. We recognise that the final judgment on which risk scores are, in reality, easy to use will lie with the end user in any particular setting. Secondly, authors of some of the primary studies included in this review were developing a local tool for local use and made few or no claims that their score should be generalised elsewhere. Yet, the pioneers of early well known risk scores49 68 have occasionally found their score being applied to other populations (perhaps ethnically and demographically different from the original validation cohort), their selection of risk factors being altered to fit the available categories in other datasets, and their models being recalibrated to provide better goodness of fit. All this revision and recalibration to produce “new” scores makes the systematic review of such scores at best an inexact science.
Why did we not recommend a “best” risk score?
We have deliberately not selected a single, preferred diabetes risk score. There is no universal ideal risk score, as the utility of any score depends not merely on its statistical properties but also on its context of use, which will also determine which types of data are available to be included.75 76 Even when a risk model has excellent discrimination (and especially when it does not) the trade-off between sensitivity and specificity plays out differently depending on context. Box 3 provides some questions to ask when selecting a diabetes risk score.
Box 3: Questions to ask when selecting a diabetes risk score, and examples of intended use
What is the intended use case for the score?
If intended for use:
In clinical consultations, score should be based on data on the medical record
For self assessment by lay people, score should be based on things a layperson would know or be able to measure
In prevention planning, score should be based on public health data
What is the target population?
If intended for use in high ethnic and social diversity, a score that includes these variables may be more discriminatory
What is expected of the user of the score?
If for opportunistic use in clinical encounters, the score must align with the structure and timeframe of such encounters and competencies of the clinician, and (ideally) be linked to an appropriate point of care prompt. Work expected from the intended user of the score may need to be incentivised or remunerated, or both
What is expected of the participants?
If to be completed by laypeople, the score must reflect the functional health literacy of the target population
What are the consequences of false positive and false negative classifications?
In self completion scores, low sensitivity may falsely reassure large numbers of people at risk and deter them from seeking further advice
What is the completeness and accuracy of the data from which the score will be derived?
A score based on automated analysis of electronic patient records may include multiple components but must be composed entirely of data that are routinely and reliably entered on the record in coded form, and readily searchable (thus, such scores are only likely to be useful in areas where data quality in general practice records is high)
What resource implications are there?
If the budget for implementing the score and analysing data is fixed, the cost of use must fall within this budget
Given the above, what would be the ideal statistical and other properties of the score in this context of use?
What trade-offs should be made (sensitivity v specificity, brevity v comprehensiveness, one stage v two stage process)?
Risk scores as complex interventions
Our finding that diabetes risk scores seem to be used rarely can be considered in the light of the theoretical literature on diffusion of innovation. As well as being a statistical model, a risk score can be thought of as a complex, technology based innovation, the incorporation of which into business as usual (or not) is influenced by multiple contextual factors including the attributes of the risk score in the eyes of potential adopters (relative advantage, simplicity, and ease of use); adopters’ concerns (including implications for personal workload and how to manage a positive score); their skills (ability to use and interpret the technology); communication and influence (for example, whether key opinion leaders endorse it); system antecedents (including a healthcare organisation’s capacity to embrace new technologies, workflows, and ways of working); and external influences (including policy drivers, incentive structures, and competing priorities).77 78
Challenges associated with risk scores in use
While the developers of most diabetes risk scores are in little doubt about their score’s positive attributes, this confidence seems not to be shared by practitioners, who may doubt the accuracy of the score or the efficacy of risk modification strategies, or both. Measuring diabetes risk competes for practitioners’ attention with a host of other tasks, some of which bring financial and other rewards. At the time of writing, few opinion leaders in diabetes seem to be promoting particular scores or the estimation of diabetes risk generally—perhaps because, cognisant of the limited impacts shown to date (summarised in table 5), they are waiting for further evidence of whether and how use of the risk score improves outcomes. Indeed, the utility of measuring diabetes risk in addition to cardiovascular risk is contested within the diabetes research community.79 In the United Kingdom, the imminent inclusion of an application for calculating QDScore on EMIS, the country’s most widely used general practice computer system, may encourage its use in the clinical encounter. But unless the assessment of diabetes risk becomes part of the UK Quality and Outcomes Framework, this task may continue to be perceived as low priority by most general practitioners. Given current evidence, perhaps this judgment is correct. Furthermore, the low positive predictive values may spell trouble for commissioners. Identifying someone as “[possibly] high risk” will inevitably entail a significant cost in clinical review, blood tests, and (possibly) intervention and follow-up. Pending the results of ongoing impact studies, this may not be the best use of scarce resources.
Delivering diabetes prevention in people without any disease requires skills that traditionally trained clinicians may not possess.80 We know almost nothing about the reach, uptake, practical challenges, acceptability, and cost of preventive interventions in high risk groups in different settings.12 The relative benefit of detecting and targeting high risk people rather than implementing population-wide diabetes prevention strategies is unknown.13 Effective prevention and early detection of diabetes are likely to require strengthening of health systems and development of new partnerships among the clinicians, community based lifestyle programmes, and healthcare funders.81
Mechanisms by which risk scores might have impact
Although most authors of papers describing diabetes risk scores have hypothesised (or seem to have assumed) a clinical mechanism of action (that the score would be used by the individual’s clinician to target individual assessment and advice), the limited data available on impact studies (see table 5) suggest that a particularly promising area for further research is interventions that prompt self assessment—that is, laypeople measuring their own risk of diabetes. The preliminary findings from the impact studies covered in this review also suggest that not everyone at high risk is interested in coming forward for individual preventive input, nor will they necessarily stay the course of such input. It follows that in areas where aggregated data from electronic patient records are available, the diabetes risk scores may be used as a population prediction tool—for example, to produce small area statistics (perhaps as pictorial maps) of diabetes risk across a population, thereby allowing targeted design and implementation of community level public health interventions.82 Small area mapping of diabetes risk may be a way of operationalising the recently published guidance on diabetes prevention from the National Institute for Health and Clinical Excellence, which recommends the use of “local and national tools . . . to identify local communities at high risk of developing diabetes to assess their specific needs.”83
Towards an impact oriented research agenda for risk scores
We recommend that funding bodies and journal editors help take this agenda forward by viewing the risk score in use as a complex intervention and encouraging more applied research studies in which real people identified as at “high risk” using a particular risk score are offered real interventions; success in risk score development is measured in terms of patient relevant intermediate outcomes (for example, change in risk score) and final outcomes (incident diabetes and related morbidity) rather than in terms of the statistical properties of the tool; a qualitative component (for example, process evaluation, organisational case study, patient’s experience of lifestyle modification) explores both facilitators and barriers of using the score in a real world setting; and an economic component evaluates cost and cost effectiveness.
Conclusion
Millions of participants across the world have already participated in epidemiological studies aimed at developing a diabetes risk score. An extensive menu of possible scores are now available to those who seek to use them clinically or to validate them in new populations, none of which is perfect but all of which have strengths. Nevertheless, despite the growing public health importance of type 2 diabetes and the enticing possibility of prevention for those at high risk of developing it, questions remain about how best to undertake risk prediction and what to do with the results. Appropriately, the balance of research effort is now shifting from devising new risk scores to exploring how best to use those we already have.
What is already known on this topic
The many known risk factors for type 2 diabetes can be combined in statistical models to produce risk scores
What this study adds
Dozens of risk models and scores for diabetes have been developed and validated in different settings
Sociodemographic and clinical data were much better predictors of diabetes risk than genetic markers
Research on this topic is beginning to shift from developing new statistical risk models to considering the use and impact of risk scores in the real world
Notes
Cite this as: BMJ 2011;343:d7163
Footnotes
We thank Helen Elwell, librarian at the British Medical Association Library, for help with the literature search; Samuel Rigby for manually removing duplicates; and Sietse Wieringa, Kaveh Memarzadeh, and Nicholas Swetenham for help with translation of non-English papers. BMJ reviewers Wendy Hu and John Furler provided helpful comments on an earlier draft.
Contributors: DN conceptualised the study, managed the project, briefed and supported all researchers, assisted with developing the search strategy and ran the search, scanned all titles and abstracts, extracted quantitative data on half the papers, citation tracked all papers, checked a one third sample of the qualitative data extraction, and cowrote the paper. TG conceptualised the qualitative component of the study, extracted qualitative data on all papers, independently citation tracked all papers, and led on writing the paper. RM independently scanned all titles and abstracts of the electronic search, extracted quantitative data from some papers, assisted with other double checking, and helped revise drafts of the paper. TD helped revise and refine the study aims, independently double checked quantitative data extraction from all papers, and helped revise drafts of the paper. CM advised on systematic review methodology, helped develop the search strategy, extracted quantitative data from some papers, and helped revise drafts of the paper. TG acts as guarantor.
Funding: This study was funded by grants from Tower Hamlets, Newham, and City and Hackney primary care trusts, by a National Institute of Health Research senior investigator award for TG, and by internal funding for staff time from Barts and the London School of Medicine and Dentistry. The funders had no input into the selection or analysis of data or the content of the final manuscript.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; and no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not required.
Data sharing: No additional data available.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.