Prescribing indicators for UK general practice: Delphi consultation studyBMJ 2000; 321 doi: https://doi.org/10.1136/bmj.321.7258.425 (Published 12 August 2000) Cite this as: BMJ 2000;321:425
- Stephen M Campbell, research fellow ()a,
- Judy A Cantrill, clinical senior lecturera,
- Dave Roberts, managerb
- a National Primary Care Research and Development Centre, University of Manchester, Manchester M13 9PL
- b Prescribing Support Unit, Brunswick Court, Leeds LS2 7RJ
- Correspondence to: S M Campbell
- Accepted 25 May 2000
Objectives: To identify prescribing indicators based on prescribing analysis and cost (PACT) data that have face validity for measuring quality or cost minimisation.
Design: Modified two round Delphi questionnaire requiring quantitative and qualitative answers.
Setting: Health authorities in England.
Participants: All health authority medical and pharmaceutical advisers in the first round and lead prescribing advisers for each health authority in the second round.
Main outcome measures: Face validity (median rating of 7–9 on a nine point scale without disagreement) and reliability (rating 8 or 9) of indicators for assessing quality and cost minimisation.
Results: Completed second round questionnaires were received from 79 respondents out of 99. The median rating was 7 for cost minimisation and 6 for quality, and in all except four cases individual respondents rated indicators significantly higher for cost than for quality. Of the 41 indicators tested, only seven were rated valid and reliable for cost minimisation and five for quality.
Conclusion: The 12 indicators rated as valid by leading prescribing advisers had a narrow focus and would allow only a limited examination of prescribing at a general practice, primary care group, or health authority level.
Quality of care within the NHS is a seminal focus of government policy. This focus on quality has driven the development of new organisational structures such as the National Institute for Clinical Excellence and the national performance framework to measure progress in six areas of health care. 1 2 In addition, clinical governance structures are being put in place to provide “a framework through which primary care groups will be accountable for continuously improving the quality of their services.”1
Prescribing indicators for general practice have been used in the NHS for over two decades3 and are likely to have a central role in the clinical governance activities of many primary care groups. This is because prescribing continues to grow at about 9% a year4 and two thirds of all general practice consultations generate a prescription.5 The national performance framework described cost effective prescribing as an important element of the “effective delivery of appropriate healthcare.”2
Prescribing is a controversial area of quality assessment.6 Previous research has highlighted the importance of critical approaches to prescribing, 6 7 defining and measuring the appropriateness of prescribing,8-10 variations in prescribing across general practices,11 adherence to standards,12 and the role of prescribing analysis and cost (PACT) data in general practice.13 Few validated quality indicators exist for prescribing in the public domain. 6 7 Avery et al concluded that “further research is needed into the development and use of indicators based on PACT.”11 The Prescribing Support Unit has developed a set of indicators based on PACT data. It advocates their use as a starting point when comparing the performance of health authorities or primary care groups with that of other authorities or groups or when comparing prescribing among general practices to identify outliers or those which are more likely to benefit from interventions to modify behaviour. PACT data are comprehensive, universal, and timely but they are not combined with diagnoses, data on specific patients, or any outcome measure. Projects are underway that aim to link prescribing and clinical data in order to produce quality indicators.
Anecdotal evidence suggests that prescribing indicators are more appropriately related to cost than quality, particularly at the practice level (the unit of analysis for most prescribing indicators). We report the findings of a two round Delphi consultation 14 15 that sought to identify which of the most commonly used prescribing indicators in the United Kingdom are face valid and reliable indicators of quality or cost minimisation. The questionnaire included only indicators of drug use that could be derived from PACT data and basic demographic features of practice populations.
Participants and methods
A list of 31 prescribing indicators was generated from on two main sources: prescribing indicators with evidence of face validity in a previous Delphi consultation7 and, most importantly, prescribing indicators used at the time of the survey by the Prescribing Support Unit.3
In May 1999 we sent the first questionnaire of a modified two round Delphi consultation to every pharmaceutical and medical adviser in England (n=305). Respondents were asked to rate each indicator against two continuous 1 to 9 integer scales: “Is this indicator a useful measure of cost minimisation?” and “Is this indicator a useful measure of quality?” Definitions of these two constructs were provided in the covering letter and questionnaire sent to respondents. Cost minimisation was defined as “using the lowest cost preparation with no adverse effect on benefit” and quality as “optimisation of health/well-being for the whole practice population.” Respondents were also asked to state whether they currently used each indicator. The questionnaire invited respondents to comment on each of the 31 indicators.
No indicators were discarded between rounds, but 10 indicators were added; two indicators related to statins and eight were minor variations on indicators used in the first round and were based on comments received in that round. The second round questionnaire therefore contained 82 ratings (41 each for cost minimisation and quality).
Participants who were sent the second round questionnaire were given three types of feedback from the first round for each indicator included in both rounds: a frequency distribution of scores (on scales of 1 to 9), a median (face validity) score for both scales, and qualitative comments (figure). Qualitative comments made during round one were transcribed and summarised. We included comments that illustrated the negative and positive attitudes expressed in the first round to provide contextual information on which respondents could base their ratings. We did not feed back to respondents their previous score.
After obtaining comments from a wide range of medical and pharmaceutical advisers (n=154) in the first round, we used the second round to achieve consensus among respondents at the health authority level. Second round questionnaires were sent in July 1999 to the lead prescribing adviser at each health authority in England (n=99) in order to obtain one single health authority return. Respondents were asked to rate each indicator using the same method as in the first round. Non-responders received one follow up mailing.
The validity data presented in this paper are based on second round median ratings only.16 We used a rating scale based on the RAND appropriateness method.16 Indicators with an overall median rating of 7, 8, or 9 without disagreement were rated face valid; indicators rated with an overall median of 1–3 and 4–6 were rated as invalid and equivocal respectively. Disagreement was defined as 30% or more scores in both the bottom (1–3) and top (6–9) tertile.17 Previous research has found that indicators rated 8 or 9 without disagreement are also reliable because higher rated indicators are more likely to be reproduced by a different panel of the same stakeholders rating the same set of indicators.18 Indicators rated with an overall median of 8 and 9 were therefore considered face valid and reliable.
Scores were analysed by using SPSS with non-parametric tests (Wilcoxon's z test) to examine whether indicators were significantly more likely to be rated valid for cost or quality.
Completed second round questionnaires were received from 79 respondents out of 99 (response rate of 79%). Overall median ratings for the complete set of 41 ratings in the second round were calculated for cost minimisation and quality. These were calculated from individual medians not from the raw scores. The median rating was 7 for cost minimisation and 6 for quality. Table 1 shows that 17 indicators were rated higher for cost, 16 higher for quality, and eight were rated identically. Overall, there was no significant difference in ratings for cost or quality (Wilcoxon's z =−0.76, P=0.45). However, in all except four cases individual respondents rated indicators significantly higher for cost than for quality (by Wilcoxon's z test). The four exceptions were ratio of compound diuretics items to all diuretic items; ratio of antibiotic items for co-amoxiclav or 4-quinolones to the number of items for all antibiotics; percentage of total net ingredient cost on drugs of limited clinical value; and percentage of non-steroidal anti-inflammatory drug items from ibuprofen, diclofenac, and naproxen.
No indicators were rated with an overall median of nine. Twenty five indicators were rated face valid for cost minimisation and 18 for quality. Of these, nine were rated valid for both (table 1). Although the remaining indicators were all rated as equivocal quality indicators, nine were rated as invalid for cost minimisation. No indicators were rated invalid for quality. Twelve indicators were rated reliable, seven for cost minimisation and five for quality.
Only two of the indicators rated valid and reliable for cost or quality in this study were currently being used by over 50% of the sample (table 2). These were generic prescribing rate and ratio of bendrofluazide 2.5 mg items to all bendrofluazide items.
Clinical governance is designed to be a means by which health organisations at different service levels can maintain and improve quality of care. Prescribing will constitute a key clinical governance objective of many primary care groups. This study aimed to discover how many of the most frequently used prescribing indicators are rated valid by advisers responsible for managing prescribing in health authorities in England. Our findings suggest that advisers believe that prescribing indicators based on PACT at the population level are less valid for quality than for cost minimisation.
Thirty three of the 41 indicators rated in the second round were found to be face valid for either cost (n=25) or quality (n=18) or both (n=9). However, only 12 indicators were also rated reliable—seven for cost and five for quality. These 12 indicators have a narrow focus and will allow only a restricted assessment of prescribing—for example, four of the seven indicators for cost minimisation relate to generic prescribing. Hence, the results obtained with these indicators need to be interpreted carefully and their limitations explicitly acknowledged.
PACT data make some, but by no means all, aspects of prescribing measurable. Three important decisions have to be made when collecting data on prescribing indicators. Firstly, what is the intended unit of analysis (for example, practice population, all individuals with a given condition, an individual)? Secondly, who is going to collect the data (health authorities, primary care groups, individual practices)? Thirdly, what are the resources required for data collection (patients' medical records or PACT)? Prescribing indicators can be used for various purposes, and it is vital for quality assessment that this purpose is made explicit.19 The validity of any type of indicator is related to its intended purpose. Additional resources are needed to produce and collect data for indicators relating to individual patients rather than populations and for indicators requiring examination of individual patients' records rather than PACT data. 9 20
What is already known on this topic
Indicators based on PACT data have been developed to allow comparison of prescribing behaviour between health authorities, primary care groups, and general practices
Little is known about the way that PACT based indicators are used in practice
What this study adds
Some PACT based indicators are currently viewed as measures of quality
Consensus about the validity of PACT based indicators was low: five of 41 were judged to be valid for quality and seven for cost minimisation
These indicators have a narrow focus and allow only limited examination of prescribing
Our finding that five of the indicators were rated face valid for measuring quality does not fit within the model of indicators proposed by Queenborough and Roberts.20 They advocated that only when prescribing and clinical data are linked can quality be measured, usually with reference to individual patient data. This view was shared by many respondents in the survey when answering an open ended question about how the quality of prescribing can or should be measured. Respondents felt that PACT indicators are process measures that need to be linked to clinical audit or patient outcome measures. This view also reflects the opinion that definitions and measurements of quality of care are most meaningful when applied to individual patients.21
Use of indicators
Our findings suggest three further caveats for people engaged in quality assessment or improvement, including primary care groups in the United Kingdom. Firstly, indicators are not measures of poor performance. Rather, they identify potential problems that may require investigation by other methods, usually audit. Secondly, it is important to be clear about what the indicators are intended to measure and what conclusions can be claimed from their use. Thirdly, for indicators to be useful for quality assessment or improvement, consistent and comparable data must be available across the relevant healthcare organisations.
Only two of the 12 indicators rated valid and reliable in this study were being used by over 50% of prescribing advisers in England. This has important implications for proponents of national sets of prescribing indicators. Prescribing indicators at the population level, such as those examined here, can never be robust enough to give any more than an absolute rather than relative measure of performance. However, our findings provide a starting point for developing a common set of prescribing indicators.
We thank Clive Jackson, director of the National Prescribing Centre, for help with sending out the questionnaire and the respondents who took part in the Delphi consultation.
Contributors: The study was devised by SMC, JAC, and DR. SMC and JAC did the Delphi consultation and analysed the data. All three authors wrote the paper, with SMC as the principal author. SMC and JAC are the guarantors.
Funding This project was funded out of National Primary Care Research and Development Centre core funding from the Department of Health. The Prescribing Support Unit is funded by the Department of Health.
Competing interests None declared.