Rapid Responses to:

ANALYSIS: Iona Heath, Julia Hippisley-Cox, and Liam Smeeth Measuring performance and missing the point? BMJ 2007; 335: 1075-1076 [Full text]

Rapid Responses: Submit a response to this article

Rapid Responses published:

Measuring performance- but not quite missing the point

Peter J Campbell   (23 November 2007)

So what are we to do

Trefor J Roscoe   (23 November 2007)

QOF - Work in progress

William J Beeby   (23 November 2007)

What do the public think?

Chris E Nancollas   (25 November 2007)

Does success mean success?

Clare A MacArthur   (25 November 2007)

Fucntion matters more than measured BP

Anthony E J Fitchett   (25 November 2007)

I profoundly disagree

L Sam Lewis   (25 November 2007)

Understanding the Effects of the QoF

Andrew L Spooner   (25 November 2007)

Observing performance and missing the point?

Graham Wheatley   (25 November 2007)

Perfomance payment and fragmented thinking

Juan Gérvas   (26 November 2007)

right and wrong foci of payment for performance

Barbara Starfield   (26 November 2007)

Work in Progress

Christopher. L. Manning   (26 November 2007)

What we measure defines who we are and what we value

Tracy Monk   (27 November 2007)

Is there an art to medicine, or is it just bulk sales?

R. Warren Bell   (27 November 2007)

It is the lack of evidence of the NET benefit of the framework itself, not the evidence around individual indicators that is the issue

Les J Toop, Dee Mangin   (28 November 2007)

Performance...who measures, and how?

Mark E McConnell   (29 November 2007)

Punished by Rewards

Paul P Glasziou   (29 November 2007)

Re: It is the lack of evidence of the NET benefit of the framework itself, not the evidence around individual indicators that is the issue

L Sam Lewis   (29 November 2007)

Measure outcomes routinely

Tim Benson, Justin Whatling   (30 November 2007)

Goodhart's Law

Peter G Davies   (1 December 2007)

Goodhart's Law

Richard Bartley   (3 December 2007)

Dealing with conflicts between the QOF and patient care

Graham N Box   (4 December 2007)

Measuring performance- but not quite missing the point 23 November 2007
Peter J Campbell, Regional Director, Quality Improvement in Health USAID ZdravPlus Project, 16 Bozbozor, Tashkent, Uzbekistan 100077 Send response to journal: Re: Measuring performance- but not quite missing the point	It is amazing to see what has happened in the UK in the last few years in the area of quality improvement, and how a unified healthcare system has been able to implement such nationwide changes. In the area of the world where I am currently working, similar but limited interventions are already showing positive results, and are leading to changes in the way healthcare is delivered. I agree with all the conclusions of the paper, but see these as minor tweaks to what is essentially an impressive and laudable (and expensive) pilot study- in global terms. I sincerely believe that in general there will be substantially improved clinical outcomes(always the most difficult to measure) observed over time as medical practitioners adapt their worldview as they implement proven treatments/ protocols. There is a balance here, between proof and individual judgements, but there has definitely been a need to swing the pendulum more towards the proof spectrum than the subjective, loosely- based-on (possibly outdated) evidence spectrum that used to be the rule. As such projects spread throughout the world, there is a need to improve these improvement initiatives. This paper highlights just this need, to the betterment of patients' lives I trust. Competing interests: None declared
So what are we to do 23 November 2007
Trefor J Roscoe, GP Principal Sothall Medical Centre, Sheffield S20 5JX Send response to journal: Re: So what are we to do	I am astounded by the article by Heath and colleagues with their criticism of a process that they themselves admit cannot yet have shown its full benefits. Early in the article they state that "Evidence based care was never meant to be a substitute for clinical judgment but, combined with the inducements of the quality and outcomes framework, it becomes so." which suggests that we should all be overriding the evidence depending on our personal whims when it suits us. They then go on to complain that there is no evidence for health improvement as a result of the QoF. Previous large scale attempts to improve management of specific conditions, such as was done in Sheffield with aspirin use over 10 years ago, have taken 5 years or more to show significant results. QoF has only been going for a few years so it is no wonder there is no evidence of improvement yet. They even admit this at the end. The use of a reference about financial incentives improving documentation but not immunisation rates in an American study from over 8 years ago is dubious. Is this actually evidence that applies to 21st Century British General Practice? When UK GPs were incentivised in the 1992 contract immunisation rates soared. I suspect that I would wish to ignore the reference and believe my clinical judgment and the experience of the last 15 years that payments for immunisation has given the UK one of the highest immunisation rates in the developed world. Later in the article is the following statement "None of the framework measures estimate clinically important outcomes. What they assess is treatment processes that are supposed to lead to improved outcomes." What is this referring to? The HbA1c level in diabetes has been shown to correlate with complication rates and thus measuring it and getting it below the recommended level is clinically important. Blood pressure levels (which have been shown to have gone down) are a measurable and important clinical outcome in a wide variety of morbidities. The negotiators who helped create the QoF used the best evidence available and were advised by experts in all the relevant fields. Are Heath et al seriously suggesting they know better than the experts? The finish by saying "Until the undoubted and now well documented increase in process is translated into tangible outcomes such as diabetes complication rates, renal failure in hypertension, or incidence of myocardial infarction or smoking related deaths, the benefits and cost effectiveness of the exercise cannot be estimated." Perhaps criticism of the process should wait until the outcome is known? The Government chose to incentivise General Practice in this way and to use the best evidence at the time to measure performance. What are the authors suggesting; evidence can be ignored, experts are wrong and GPs should forgo one of the largest investments in General Practice in the last 20 years? How will doing this benefit patients? My clinical judgment tells me that what I am doing for my patients is improving their health so I will continue to do it, despite the current lack of evidence and the criticism of the process by Dr Heath and her colleagues. Competing interests: A GP with a high QoF score and patients who are benefitting
QOF - Work in progress 23 November 2007
William J Beeby, GP Middlesbrough TS8 0RA Send response to journal: Re: QOF - Work in progress	The authors claim that we are missing the point and not showing results, whilst distorting care in favour of those disease areas fortunate enough to have been selected for inclusion in the QOF. I don't think this reflects the entire truth, though there will inevitably be items in the QOF framework that appear to be micro-management of a practices work in the extreme, and performed because there is a financial incentive. It is very early in the days of QOF to expect to see dramatic results, though I am confident that todays effective management of hypertension will result in reductions of progression to IHD and heart failure in the future. However to suggest (for example) that the only thing we do with hypertensive patients is measure their blood pressure whilst ignoring the more holistic elements of their care would be totally wrong. The QOF will evolve as evidence of what constitutes best practice changes over time. There has never been any suggestion that it was perfect at the first go, but it represents a significant attempt to produce accurate measures of certain aspects of our care of major diseases. There will always be debate about whether measuring a BP or an HbA1c is a task oriented tick box or an outcome. It depneds on the result of course, and a good reading is a successful outcome. The QOF is but part of our work and since much of the rest cannot be placed in segments to be measured does not detract from the importance of it. Let the QOF evolve and be a little patient for some of the outcomes to become clearer. If practices are doing some of it because of the financial reward, then so be it. You wouldn't want to reward then for doing work that was not supported by an evidence base, would you? Competing interests: None declared
What do the public think? 25 November 2007
Chris E Nancollas, GP Newnham, Glos GL15 4TX Send response to journal: Re: What do the public think?	Dear Ms Godlee, The authors are to be congratulated for articulating the concerns of some general practitioners about the whole QoF business. Two things in particular stand out. The first is the general assumption that the QoF is a good thing because patients want to live longer. I share the authors opinion that this is questionable. The general public are considerably more thoughtful than we give them credit for, and increasing numbers of my patients and friends openly question the wisdom of surviving into their late eighties and nineties. Secondly, the QoF has made the computer the focal point of the consultation. This has partitioned the profession into the computer literate and 'others'. I would just like to point out that the best doctor I ever consulted was definitely one of the 'others', and I think there is a real danger that we are confusing keyboard skills with competence. Responding to computer prompts is not consulting, and we could be on a very slippery slope here. Yours sincerely, Chris Nancollas Competing interests: GP with high QoF score
Does success mean success? 25 November 2007
Clare A MacArthur, Lecturer Practitioner in Diabetes University of York YO10 5DD Send response to journal: Re: Does success mean success?	Heath et al. (2007) make some valid points about the potentially limiting effects of the QOF on care. Where the QOF does not contribute to effective care, paradoxically, may be in the areas that it is seen as a success; whereas 'the higher the better' might hold true for immunisation rates, 'the lower the better' does not always hold true for measures such as blood pressure, as the authors point out, and HbA1c levels. Low HbA1c results may indicate frequent, possibly asymptomatic hypoglycaemia and this in turn leads to an increased burden on people with diabetes, relatives, emergency services, A & E departments and even inpatient facilities. The thoughtful, skilled practitioner will bear this in mind when caring for people with diabetes, but will practitioners choose to develop their skills if they are achieving high scores for diabetes within the practice already? At practice level, will a paramedic call-out to attend someone with severe hypoglycaemia be recognised as avoidable, or just reviewed as a new problem? Generally in the QOF, the process outcomes are not set at 100% achievement, so there is indeed some scope for partnership working with patients, and professional judgement, but providing good care will always require more than can be measured. Competing interests: None declared
Fucntion matters more than measured BP 25 November 2007
Anthony E J Fitchett, General Practitioner Mornington Health Centre, 169 Eglinton Road, Dunedin, New Zealand Send response to journal: Re: Fucntion matters more than measured BP	Trefor Roscoe feels that lowering blood pressure (BP) is a laudable outcome in itself: my elderly patient who cannot function vertically if her BP is lower than about 170/95 might disagree. She might just feel that the ability to lead a normal life, despite an "unacceptable" BP, is a better outcome than a blood pressure that is officially "acceptable" but which makes normal life impossible. And I might agree with her. Competing interests: I am a GP, subject to a battery of Performance Management Indicators which at present do not include BP measurements
I profoundly disagree 25 November 2007
L Sam Lewis, GP Surgery, Newport, Pembrokeshire SA42 0TJ Send response to journal: Re: I profoundly disagree	As a GP who has willingly embraced the New GMS contract, I am fast becoming inured to carping and whingeing from outflanked or incompetent DoH and NHS managers.  But when my erstwhile colleagues join the chorus, I am moved to reply.   Was it not the staple fare of the last 20 years for all the College clique, that quality in General Practice was varied and in need of improvement ?   Did we not all agree that an incentive scheme should reward  sound, proven and demonstrable quality of care ?   Now that GPs have been clearly shown to  ‘overperform’ why have our ‘own’ ( Heath, Hippisley-Cox, and Smeeth )  turned upon us ? It does indeed behove us ALL to think critically about what we are doing. Evidence of improved quality of documentation is matched with CONSIDERABLE evidence of improvement in underlying standards of care, and outcomes.  The website attests to this ‘The Quality and Outcomes Framework (QOF) is a voluntary annual reward and incentive programme for all GP surgeries, detailing practice achievement results. It is not about performance management but resourcing and then rewarding good practice.    Validation of achievement , and modification of scheme parameters is clearly the responsibility of NHS managers.   There have undoubtedly been useful achievements, say the authors, grudgingly:- “However, the system is in danger of missing the point of both quality and general practice” .  No it isn’t.   QOF is entirely voluntary , and can be ignored by any GP who wants to practice homespun and traditional folklore remedies, or forgo rewards of filthy lucre for a place in heaven or their patient’s hearts.  And anyone who espouses QOF is encouraged to take account of  defensible clinical or consensual exceptions from any parameter. The clinical activities of QOF are ‘largely’ evidence based, begrudge the authors, but  “almost all interventions cause some harm, and even when effective treatments are applied to a series of patients in clinical practice some will be harmed (although more will benefit)”    But of course – Isn’t this true for every known element of medicine or surgery ??  But they then wander into unevidenced unreason. “The risks of harm tend to increase with age, as does the potential for benefit. The stakes therefore become higher as the evidence becomes more tenuous because many trials focus on younger patients.”   The evidence-base for QOF performance measures is considered, and parameters are agreed by a National Committee of experts.  Go plead your case to them. “Evidence based care was never meant to be a substitute for clinical judgment” -  but it should be a substitute for unevidenced and arbitrary whims of  the unexamined practice.. “Mechanistic..  blanket..  embedded into computer.. fixed.. static.. and stifled..  Interventions become routine.”   Well, they certainly can be, be only if you let them.  In what might otherwise be tough drudgery, I enjoy and take heart from demonstrating effective improvement in my patient’s care ! “Practitioners are no longer required to grapple with the innate uncertainty of each different clinical situation…” ?  Yes they are – the patients see to that !  The authors suggest that “currently no robust systems in place to measure or monitor  [ subgroup harms]. I put it to them that the monitoring systems of the QOF provides the most comprehensive and auditable robust system yet available.   Far from “diminish[ing] the responsibility of doctors to think”  QOF focuses the GP team on demonstrable quality improvement.   If there are drawbacks , then let us change the quality measures !  For example, the authors again repeat that “the failure to make any allowance for age means that doctors are encouraged to overtreat hypertension in old people4 with the danger of causing fainting, falls, and fractures”.  It ought to be a simple matter to furnish the QOF committee with the evidence for this , and persuade them to include an Age cut-off for the Hypertension domain ( and thereby further justify QOF ) …   Or perhaps the authors will concede that  there are already many patients who are being denied effective an powerful healthcare interventions , simply because they are ‘old’ ?. Although QOF  “necessarily concentrates on clinical activities that are easily measured”  it must also meet  SIGNIFICANT HEALTH AND EVIDENCE-BASE criteria. The authors cite “ hugely increased prescribing of some drugs. Almost all of this is designed to prevent future events rather than alleviate present suffering.”   Well I hope so !   Indeed, on all the evidence mustered, it jolly well ought to be so. “Prevention has its merits, but was this an intention of the framework and what are the opportunity costs for other healthcare interventions?”   Yes , Yes, and considerably less than most QOF interventions, I reply.   Let the Professors submit any contrary  evidence to the Committee. As to tackling health inequalities, QOF has ‘the potential to work in the opposite direction’.   That seems perverse.  QOF is programmed to identify all those who might benefit ( the Domain Register), and then measures whether they are getting an intervention, permitting a GP to document valid reasons for not intervening.  Perhaps we should make it more universal, with compulsion ?  “Most fundamentally, it encourages the illusion that health inequalities can be solved by the health service” – No, except where that has been researched and evidenced, and consensus achieved as to its importance in healthcare. “It is based on the astonishing assumption that everyone wants to live as long as possible”  - No it isn’t.  It permits any patient to withhold consent to any treatment, and encourages this in ‘exception recording’ .  A major valid criticism of QOF has been the inconsistent, and highly variable use of exception reporting.   “The poor are much more likely to experience co-morbidity”.  If so then QOF encourages the practice team to approach them from multiple directions.  Lowering a BP, or a Cholesterol, will be multiply rewarded and hence more encouragement is provided for Co-morbid/poor people to get the healthcare they need..   The authors then degenerate into inconsistent and contradictory positions on “most marginalised people, who have a combination of physical and mental illnesses often compounded by drug or alcohol dependence” claiming that they are difficult to engage in health care and even more difficult to coerce into the framework's unitary care pathways.   Actually QOF simply rewards Practices who make the effort to identify these people, and encourages doctors to offer them help.   There is no coercion, and no ‘unitary care pathway’ that I am aware of.  “Patients living in adverse social circumstances are also more likely to be taking maximal tolerated therapy without achieving the desired outcome. Not surprisingly, such people are much more likely to be reported as exceptions and, once given that status, are at risk of receiving proportionately less attention.“   Phew !   How to face both ways and inside-out in one proposition ? The incentivisation itself is worthy of further discussion.  “As people living in deprived areas are sicker, more therapeutic effort will be needed for doctors to reach fixed targets.”   Professor Pringle raised  this issue, early on in the discussions  “All agree that we need to measure the quality of health care, including the care given by individual doctors. Measuring "goodness" requires accurate data used appropriately, and it must be done without demoralising and demotivating staff. Do current measures fulfil these requirements, and if not, what measures should be used? “ (BMJ 2002;325:704-707) This means that doctors working in [deprived] areas MAY have to work harder to achieve the same remuneration.  But QOF claims that “ It is not about performance management but resourcing and then rewarding good practice.”    I agree that  increased payments for higher disease prevalence do not take severity and complexity into account, and worse, the perverse ‘square-root correction’ ( which the authors do not mention) penalises high prevalence areas severely, whilst artificially rewarding low-prevalence, well-heeled areas.  This disincentivisation to working in poorer areas needs urgent correction using appropriate measures of  need combined with ‘needed effort’ to support a form of ‘reward for appropriate effort‘, rather than ‘reward for achievement’.   Thus a case is made for rewarding and resourcing the delivery of healthcare , rather than the health outcome itself.   But  the authors go on to contradict this logical thrust with “None of the framework measures estimate clinically important outcomes. What they assess is treatment processes that are supposed to lead to improved outcomes (my italics). A marked discrepancy exists between the likely effect on health and the level of monetary reward, and there seems to have been no attempt to align the two”.  Oh dear. Where quality can be measured, and there is national consensus on its importance and evidence-base, then it can be included in QOF.   If  what matters can’t be  measured, then the Global Sum funds our efforts “to provide all due an necessary primary care to those ‘who are ill, or think they are ill’” .  Thus by definition the core  nGMS contract leaves out preventive care, further justifyng the additional QOF.    Since it is voluntary ,   “authentic dialogue between doctor and patient” need not be disrupted, and no doctor need feel “fundamentally compromised”.     In Summary Measurable differences do not necessarily translate into meaningful differences in patients' lives, but they will if they are well-evidenced and appropriately encouraged through QOF Clinical care needs to be tailored to individual patients rather than using a mechanistic approach, and hence the QOF Exception-reporting facility must be stoutly defended and strengthened The quality and outcomes framework needs to include clinically important outcome measures, without diminishing the incentivisation and resourcing of the healthcare efforts required. It should also include mechanisms to measure and monitor ACTUAL harms   Competing interests: I am a GP who has benefitted from nGMS and QOF
Understanding the Effects of the QoF 25 November 2007
Andrew L Spooner, GP Grosvenor Medical Centre, Crewe, CW1 3HB Send response to journal: Re: Understanding the Effects of the QoF	The article has the wrong target. It is complaining against evidence based medicine and the way it is applied to populations. QOF is a mechanism to resource and encourage change. Learning would be assisted by understanding why the QoF had particular effects rather than criticising the whole scheme for being an effective implementation tool for interventions the authors don’t like. Whether that change is beneficial depends on the choices made in setting the structure and choosing the criteria. Politically it is difficult to justify the budget for a service that demands the right to ignore the evidence. No one said the QoF was instead of clinical care, good General Practice does both. The evidence presented, that care of deprived populations improved but it is more effort, would suggest it is a good mechanism to reduce the health divide. As the authors point out doctors avoid high work and low income but this applied before the QoF; the only difference was the lack of measurement. This could also be interpreted as a targeted improvement in care revealing the true incidence of disease and providing a reason to increase resources. The funding for care of deprived populations is a different issue. Currently it is loosely linked to disease prevalence but deprivation could be rewarded in the capitation payment mechanism. The article makes no attempt to understand why the QoF (and the underlying evidence) produced these effects. This would have helped construction of future schemes in different clinical areas. Investigation of a precursor scheme showed 5 drivers of GP activity1 1) Improved patient care - each individual criterion stands or falls on its ability to improve patient care 2) Retained autonomy - Measuring outcomes and leaving autonomy of method to the clinicians 3) Professional pride - Endorsement of the framework as a clinical entity by the professional 4) Resources - The structure of the scheme provided resources 5) A task they were going to have to do anyway - Government involvement indicated that these disease areas were their priority Quality schemes are implemented by professionals. The effect of any scheme depends on how those professionals view what they are being asked to do. QoF was successful as a scheme and as criteria because it gave these and encouraged practices to find their own methods (process) to complete tasks. Any changes should take into account the effect on the professionals as well as clinical evidence. It is the failure to persuade the authors that the current criteria operate in the interests of patients that is at the heart of this article. It is possible to look at particular schemes and criteria and map whether they are likely to be successful against the five drivers. QoF was successful because most GPs felt it was helping them and improving care. Subsequent guidance and changes have introduced more process and resources have been removed. As the authors point out resources for deprivation do not follow need. This is likely to negate some drivers and reduce the level of implementation. These areas are relatively small compared to the totality of QoF hence the success of the scheme. The good news for patients is that there are limits to what GPs will implement. Where criteria are introduced that GPs cannot be convinced improve patient care they will not be fully implemented. This would apply to some of the examples listed. GPs cannot simply be bribed, there are other drivers. The bad news for GPs is that the NHS will blame them for doing what is suggested here and making decisions about whether to implement protocols inappropriate for their population. When, for example, they do not like over treating hypertension or using mechanistic tick lists for depression their organisation will be changed to force implementation of process. Protocol based medicine is very strong. The protocol rather than the QoF would have to change if the authors are correct. The discussion of the QoF has concentrated on whether the whole framework is good or bad. Like any successful medicine or intervention it has good and bad points. The aim should be to understand why it worked, increase the benefit and minimise the problems. The QoF brought new resources to GPs and allowed care of whole diseases to shift to General Practice. It facilitated system development in the less developed practices. There will be other areas where it could be used to resource the shift of care and demonstrate quality but if process based criteria are required difficulties will arise. Designing performance indicators is difficult and they should be targeted correctly. The views of the professionals expected to implement them are important and affect what can be done. Reference 1. 1 Spooner A, Chapple A, Roland M (2001) What makes British general practitioners take part in a quality improvement scheme? Journal of Health Services Research and Development. 6: 145-50. Competing interests: NHS Confederation Negotiator for the QOF
Observing performance and missing the point? 25 November 2007
Graham Wheatley, GP Munro Medical Centre, West Elloe Avenue, Spalding PE11 2BY Send response to journal: Re: Observing performance and missing the point?	Heath et al repeat here their well-know views on the new GMS contract. They attack it for apparently replacing clinical judgement with blunt targets that measure processes rather than outcomes - all under the corrupting influence of financial incentives. Of course, the truth is very different. The quality and outcomes framework (QOF) of the new GMS contract doesn't encourage treatment of the otherwise well or vulnerable, as there are explicit exclusions that are used. There are no financial incentives to ignore clinical judgement, quite the opposite - clinicians are prompted to consider evidence-based outcome measures, so promoting the opportunity of improved clinical decision-making. Attacking targets for being aimed at processes rather than outcomes ignores the irregular incidence of significant outcomes in practice populations. It's the same twisted logic that would argue that its pointless driving safely at 30 mph in town because individual drivers are unlikely to see a reduced fatal accident rate - we do it because collectively it produces better outcomes, as predicted by credible evidence. However Heath et al assume the worst of their colleagues' efforts. Improved performance is merely "apparent". Prevention, long the mantra of those bemoaning the focus on treating acute illness, now merely "has its merits" now that GPs are now effective at promoting it widely. There is a sense of financial resentment and self-loathing in the arguments, together with a nostalgia for a golden age now gone. However, nostalgia is never what it used to be. The reality today is that GPs can be extremely proud that there's the best-ever application of evidenced - based medicine in general practice, in a system that does not financially penalise decision-making appropriate to individual patients. It's a shame that Heath et al align themselves with the ill-informed and resentful sections of the media and spend their considerable efforts attacking rather than celebrating GP's achievements. The hunt is on - what could possibly be in the other half of Heath et al's half empty glass? Competing interests: GP in the NHS
Perfomance payment and fragmented thinking 26 November 2007
Juan Gérvas, general practitioner Buitrago del Lozoya (Madrid) Spain, 28230 Send response to journal: Re: Perfomance payment and fragmented thinking	Dear Sir: The Quality and Outcomes Framework is a payment-performance type of incentive, based on the fragmented thinking of specialists which is incorporated in many protocols. Empirical evidence is required to show that overall health of patients is improved and that biological views typical of specialists are not promoted (1). Heath et al (2) make an important contribution to the literature on the topic of process versus outcome. Process should be only a proxy and cannot substitute outcomes. The time has come to abandon individual diseases, diagnosis and performance as the focus of medical care and think more on patients and their health (3-5). 1. Gérvas J, Starfield B, Violán C, Minué S. GPs with special interests: unanswered questions. Br J Gen Pract. 2007;57:912-7. 2. Heath I, Hippisley-Cox J, Smeeth L. Measuring performance an missing the point. BMJ. 2007;335:1075-6. 3. Rosemberg CE. The tiranny of diagnosis: specific entities and individual experience. Milbank Q. 2002;80:237-60. 4. Tinetti ME, Fried T. The end of the disease era. Am J Med. 2004;116:179 -85. 5. Fleetcroft R, Cookson R. Do the incentive payment in the new NHS contract for primary care reflect likely population health? J Health Serv Res. 2006;11:27-31. Competing interests: None declared
right and wrong foci of payment for performance 26 November 2007
Barbara Starfield, Professor The Johns Hopkins University Send response to journal: Re: right and wrong foci of payment for performance	Payment for performance is antithetical to patient centered care, Oriented towards professionally defined specific diseases (rather than to health problems that people face), it pays little attention to peoples’ problems as they experience them. Many of the criteria that justify the choice of measures have limited clinical benefit and/or small magnitudes of effect. Measured activities crowd out unmeasured ones. The most pernicious result of this is that populations such as children are systematically disregarded, because their problems are unrepresented among the chosen ones. As the recognition of patients’ problems (not the accuracy of diagnosis) is the rate-limiting step in achieving quality of care, it should be the most heavily rewarded, yet it never is. Commercial interests in promoting tests, procedures, and medication are now driving the choice of ‘diseases’ for quality rewards—with very uncertain benefit to overall health and distribution of health in populations. What receives no attention may be much more important than what does, as Heath and colleagues imply in their excellent summary of the subject. Competing interests: None declared
Work in Progress 26 November 2007
Christopher. L. Manning, CEO Primhe Teddington Send response to journal: Re: Work in Progress	Dear Editor, I have read all the Rapid Responses. However, the QoF essentially attempts to apply reductionist and Newtonian mechanistic approaches (good for landing a rocket on the moon) to a complex system and complex organisms living and working within that system (The NHS and humanoids within it ). We need both approaches (at least)? Good whole-person-approach GPs are everywhere; I know I have one. However, I do believe, as a result of hearing those dreaded narrative 'anecdotes' from many GPs and friends (who like me engage the NHS as patients), that many are frustrated by a sense of determinism in all of the incentivised behaviours. How can we be so sure that other non-person centred practitioners are working to a brief that essentially, at least in QoF, allows only marginal access for a person's wellbeing and whole health to enter the consultation? Resistance to the acquisition of those human and psychological skills that would help many GPs to deal simply with distress, let alone illness-caseness, is rife. Why else would we need to be simply flying in additional workforces of psychologists and not, in any way, seeking to skill-up those already at the front-line? It is also almost as if we are needing to lever the human condition back into many GP's surgeries organ-system by organ-system. I know because Primhe joins others in lobbying for various 'conditions' that are not so amenable to reductionist, mechanical approaches in those leading chaotic, disorganised and deprived lives. The RCTs that inform the QoF do not usually include such people, or those with comorbidities and or medically unexplained symptoms. And what do we do about those GPs who wish to practice according to the science, biology and evidence for how we are all "wired for health" and where the evidence for so doing is simply not currently acknowledged by the QoF mechanism or framed within its evidence- base. we could all have predicted that the pancreas, arteries and heart would have got in there first. Well, it can't stop there - the brain and mind are on the agenda and coming to a cinema near you. We can be non- fluffy and put in some very hard science there too. Indeed, the evidence for the effects of incentivising that brain in the well-chosen ways that we do and for the effects of deprivation and inverse care, are part of that evidence-base. QoF is constantly under review admittedly, but can be ignored in areas that are of no personal interest to practitioners, or those where the importance of interventions (eg Early Intervention for Psychosis and Dementia) have been established with an evidence-base, but have not been levered in yet? What happens to the people with those problems meanwhiles? The evidence currently is that we are still doing too little too late and I am not convinced that a single GPS satellite (QoF) is going to be enough to give us the 'fix' we need on these more complex issues. As for 'informed consent'. What does the more elderly or frail person do about having a discussion about being placed on a statin? You surely must all know that many such people feel absolutely beholden to their doctors and would not dream of questioning them? And where is "building therapeutic rapport" in the QoF and what would happen if it were? Yours Sincerely, Dr Chris Manning www.primhe.org Competing interests: Primhe receives funding from the Department of Health, Servier, Lundbeck, Lilly, Sanofi Aventis and Wyeth. Its relationship with these pharma companies are negotiated through the ABPI Mental Health group
What we measure defines who we are and what we value 27 November 2007
Tracy Monk, General Practitioner Vancouver, British Columbia, Canada Send response to journal: Re: What we measure defines who we are and what we value	Many thanks to Heath et al for having the courage to question quality as defined by the QOF and for laying bare the potential for harm to patients and health systems that may result from the best of intentions. Auerbach et al made a related observation about the importance of acknowledging the potential harms of some quality initiatives in the NEJM of August 9, 2007: “Just as in the rest of medicine, we must pursue the solutions to quality problems in a way that does not blind us to harms . . . or delude us about the effectiveness of our efforts.” (2) While many GP’s in other countries like Canada, applaud efforts in the UK to place fair value on the complex work of General Practice, we watch with growing discomfort some of the potential adverse consequences of the QoF. Tim Reynolds, a UK lipid clinic director, made the following troubling observations in a recent BMJ posting: “Many elderly patients are referred to my lipid clinic because they do not meet the Government targets set in the QoF despite high doses of statin and the consequent myalgia. Frequently after a discussion of the meaning of risk, these patients opt not to be treated because the likely benefits are so small they do not outweigh the adverse side effects.” “The use of risk-based cut offs . . . means that treatment cannot be avoided by doctors who have been enslaved by the target culture.” Reynolds concluded that “the main casualty of target-based medicine is common sense.” (3) As Clare McArthur (Lecturer /Practitioner in diabetes) has wisely pointed out above, in her comment titled:Does success mean success? “Whereas 'the higher the better' might hold true for immunisation rates, 'the lower the better' does not always hold true for measures such as blood pressure, and HbA1c levels.” Recent events around glitazones highlight McArthur’s point and underline the potential for adverse consequences that can come with a focus on surrogate markers and process measures, which seem to be a key component of the QOF. Clifford J. Rosen, chair of the FDA advisory committee on rosiglitazone made the following comments writing online for NEJM: "We urgently need to change the regulatory pathway for drugs for the treatment of type 2 diabetes to make clinical outcomes, not surrogates, the primary end points," Rosen noted that the drug had “caused undue harm to patients” and that the FDA had approved it “prematurely and for the wrong reasons”. He noted that among the studies evaluated, two of the largest "failed to find a significant reduction in cardiovascular events even with excellent glucose control." (4) - Prevention should be targeted where it makes both common sense, and a meaningful difference in the lives of patients, rather than just a measurable difference in a data repository. - Placing fair value on the integrative work of general practice should acknowledge the complexity of our practice more broadly than a small disease-focused subset. - Quality should be defined in a manner which reflects the importance of the key attributes of strong primary care: continuity, coordination and comprehensiveness. At the end of the day, what we measure defines who we are and what we value. The Health- Related Quality of Life of our patients is a measure that matters. Perhaps it is time to make sure that our definitions of quality in primary care capture more accurately what we all value. The authors have pointed out that the QoF initiative is “unique worldwide and is attracting considerable international interest. It therefore behoves us to think critically about what we are doing.” Yes it does. Heath et al’s efforts to do so are appreciated and as Canadian scholar John Ralston Saul has noted “Doubt is the only human activity capable of controlling the use of power in a positive way. Doubt is central to understanding." __________________________________________________________ REFERENCES 1) Heath I, Hippisley-Cox J, Smeeth L. Measuring performance and missing the point. BMJ. 2007;335:1075-6. http://www.bmj.com/cgi/content/full/335/7629/1075 2) Auerbach, Landefeld, Kaveh. Shojania. The Tension between Needing to Improve Care and Knowing How to Do It. NEJM Aug 9, 2007; 357:6 http://content.nejm.org/cgi/content/full/357/6/608 3) Reynolds, T. Prevention may not be beneficial. bmj.com, 11 Aug 2007 http://www.bmj.com/cgi/eletters/335/7614/285 4) Chair of FDA's Rosiglitazone Advisory Panel Offers Perspective on Its Decision. Physicians First Watch http://firstwatch.jwatch.org/cgi/content/full/2007/809/1 Competing interests: None declared
Is there an art to medicine, or is it just bulk sales? 27 November 2007
R. Warren Bell, Family physician Salmon Arm, BC Canada V1E 4S2 Send response to journal: Re: Is there an art to medicine, or is it just bulk sales?	I was heartened to read the remarks of Heath et al in commenting on the QoF process in Britain, which is very similar to the Chronic Disease Management (CDM) process in BC, the western-most Canadian province, where I live. I have eschewed participation in CDM activity, for the same reasons Heath et al question QoF. My principal concern is this: are we measuring outcomes because they are easy to measure, or because they are valid indicators of quality of life? Both QoF and CDM operations rely on defined endpoints that are simplistic, and often separative, i.e. diabetics and hypertensives are identified and given drugs, while people who are miserable because they are socially disadvantaged, through no fault of their own, are right off the QoF/CDM radar. And there's absolutely no room for people who are well, and want to remain that way for a very long time with a little help from their friendly GP. A second concern is that the interventions mandated within both processes seem to revolve around drugs, drugs and more drugs (actually, rather an astonishingly large amount of drugs). What about exercise? What about enhancing family support systems? What about non-pharmaceutical, biological remedies? What about meditation (tons of studies show it reduces BP)? What about making healthy eating easier in a community setting? Where would "shutting off the Broad Street pump", generally reckoned to be rather a good thing, figure in either system? Answer: what Broad Street pump? (John Snow wouldn't have earned a penny under the QoF program of his day, of course.) Finally, correcting "bad" behaviour on the part of aberrant, presumed unintelligent and backward GPs seems to be the underlying goal of both systems. Trust a 21st century band of policy-drenched medical organizers to choose to correct such behaviour with computers and money, rather than personal contact and the use of community-embedded resources. One day, perhaps, one or other of these QoF/CDM lads, laptop computer and statistical manuals in hand, will penetrate into the postulated hinterland of medical medievalism, and find, say, a kindly GP treating her patients with love and affection, common sense, practical wisdom and forbearance. And oh dear oh dear....she'll be just dreadfully behind the times. Of course, I'm sure he'll want to take her severely to task -- and rightly so! She's upset his applecart, and deserves the direst punishment for her temerity. I'm all for evidence. I'm all for evidence-based medicine. I just prefer to use ALL the evidence, and not just the simplistic, "low-hanging fruit" sort of evidence. Sometimes the plum at the top of the tree has the most interesting taste. Competing interests: I'd like some, but can't seem to find any.
It is the lack of evidence of the NET benefit of the framework itself, not the evidence around individual indicators that is the issue 28 November 2007
Les J Toop, Professor of General Practice University of Otago, Christchurch, Dee Mangin Send response to journal: Re: It is the lack of evidence of the NET benefit of the framework itself, not the evidence around individual indicators that is the issue	Earlier his year we were on study leave in the UK and were asked (by the other journal) to write a piece on QOF(1). The more we read, the more we spoke to general practitioners, the more dispirited we became. The mixture of supportive wisdom and self righteous defensive anger in the rapid responses to this excellent piece by Iona Heath et al. echo the range of views we heard. Clearly (to our jaundiced eyes looking down the barrel of a NZQOF in the making) there seemed to be those (often who have been around a while) with the wisdom and insight to see QOF for what it is: an unfortunate and far reaching ideological experiment based on P4P which has little or no rigorous evidence base on harms to balance the self evident “you get what (little) you pay (lots) for.” We would argue a simplistic and flawed system which damagingly skews the value of measurability over meaningfulness (and ignores that they are so often inversely related), an external, top down imposed system of bribery that has transformed the workload and capacity of general practice teams. A system which comes with inevitable and worryingly unquantifiable opportunity costs, both in time now unavailable to address patient defined concerns and an even more worrying and equally unquantifiable incremental loss of professional identity. A system based on physician, and through them, patient coercion, and / or if that doesn’t work, we were repeatedly told, gaming through exception reporting. The result will be a perceived reduction in the need, if not (yet) ability, for individual critical thought which will as a result be inevitably diminished - to the collective patients’ disadvantage. And then there were those who are doing very nicely out of QOF thank you very much, so defensive and so very secure in their rationalisation / justification that they are doing a great job and that clever folk have ensured that all the indicators are so evidence based that (some of or all of?) their patients must overall be better off. Thankfully, we also met many thoughtful GPs who, whilst seeing some gains, are increasingly troubled and unsure of the net benefit of the system, both to them as physicians and to their patients. It would be good to hear more from the worried fence sitters in these columns. We hear from afar murmurings that the stellar success in achieving targets and the resulting budget blow out may result in the process being further wrestled from those actually charged with doing the work. The title of our piece was ”QOF what have you done to yourselves?” hopefully our next piece (if we were ever asked again!) will be “QOF we were wrong” rather than “QOF, we told you so.” Rather than worrying about Iona's glass being half full or half empty, a collective trip to the professional opticians for re refraction might be in order for a whole new set of glasses - without the rosy tint perhaps? Listen to luminaries like Heath, Hippisley Cox and Smeeth, don’t berate them. When they are so worried, you should be too. Reference 1 The Quality Outcomes framework: What have you done to yourselves? Mangin D, Toop L BJGP 2007;57:435-7 Competing interests: Long term advocates for and providers of evidence informed professional education
Performance...who measures, and how? 29 November 2007
Mark E McConnell, Staff Internist LaCrosse, WI USA Send response to journal: Re: Performance...who measures, and how?	Heath et. al. echo sentiments I share in the US. No one should feel that pay-for-performance is unique to the UK. Once a concept has it's own acronym (P4P) I think it is safe to say it is here to stay. I suspect it is safe to also say that all that have commented share a common goal of improved patient care. When we speak, however, of "performance" then we must look at who defines this and who measures this. Hayward has written a thought provoking editorial (1) outlining potential problems that can occur when we try to pay clinicians for performance. One of these is that payors seem to want an easily quantifiable metric to use...that makes us easier to evaluate in the macro sense. As Hayward points out, it would be optimal to keep quality improvement and cost containment separate but when we introduce financial incentives for clinicians (most of whom are human) we raise the risk of gaming systems and "incentives not to care for sick people" as they can alter ones "report card". Unless there is a system in which patients are the true "customer" (payor) it is unlikely that measuring clinicians and paying for performance will go away. And I do not think it is likely we will see such a system...we certainly do not in the US excepting for the very wealthy. As such, clinicians must decide whether or not they are going to work for pay or work for patients. When I am faced with a choice between "treating the numbers" and treating the patient professional ethics take over...and each of us has a choice, a conscience, and a patient in front of us. I would love to see B4P (best-for-patient) replace P4P and yet this will be an internally guided process...not an external one. (1) Hayward,RA, Performance Measurement in Search of a Path. NEJM. 356:9 3/1/07. pp 951-953 Competing interests: None declared
Punished by Rewards 29 November 2007
Paul P Glasziou, Professor, Evidence Based Medicine University of Oxford Send response to journal: Re: Punished by Rewards	Heath et al are right to point out the potential downsides of such financial incentives that attempt to move GPs from "patient-centred" to "state-centred" care. Though I am heartened by the letters from other GPs suggesting they have enjoyed demonstrating improvements in some processes care, I would urge them to consider whether the evidence of patient benefits in outcomes is worth the cost in deprofessionalisation. Some questions we should ask are: 1. Was QOF needed to improve care? One implication of the QOF is that GPs were not making appropriate improvements in practice beforehand. This is both insulting and wrong. Data on coronary heart disease, asthma, and diabetes show considerable improvements in care were occuring well before the QOF was introduced[1]. 2. Do financial incentives in general "work"? In a systematic review of the impact of financial incentives, Petersen et al found 17 controlled studies[2]. Overall most of these showed small apparent improvements in care, but at least some of this could be attributed to better documentation rather than better processes of care. The studies also indentified some unintended effects of financial incentives such as gaming of the system and selection of patients for care, but little attention had been given to these. Only one study (of nursing home care) addressed cost- effectiveness. I wonder what NICE might have concluded if asked to assess the effectiveness and cost-effectiveness data of QOF as a health care intervention? 3. Are we confident that the items and targets set by QOF are what our patients want? As a recent editorial[3] pointed out "In an era that values respect for automony, it beggars belief how we could have arrived at a point where the very nature and content of the doctor-patient encounter is prescribed by the state." If we are to diminish individual autonomy, community informed consent would seem a minimum requirement, e.g, we might be to use community juries who are informed of QOF proposals, and only accept those that the clear majority of patients would want[4]. 4. Do we really want to move from internal professional motivation to do a good job of care to an externally imposed set of targets? Research in education suggests rewards have short term benefits but long term harms: children subject to rewards stop learning as much as possible but instead learn only what is rewarded. Similar concerns have been appropriately raised for health care[5]. Toop and Mangin rightly ask in their rapid response "It is the lack of evidence of the NET benefit of the framework itself, not the evidence around individual indicators that is the issue". I would agree. QOF seems to be large scale and expensive experiment with protocol, no informed consent, no control group, no measurement of the adverse effects, and hence no measure of the net effects. Unfortunately, uncontrolled experiments don't seem to require ethics committee approval. References 1. Campbell SM, Roland MO, Middleton E, Reeves D. Improvements in quality of clinical care in English general practice 1998-2003: longitudinal observational study. BMJ 2005;331:112-13. 2. Petersen LA, Woodard LD, Urech T, Daw C, Sookanan S. Does Pay-for- Performance Improve the Quality of HealthCare? Ann Intern Med. 2006; 145: 265-272. 3. Mangin D, Toop L. Quality and Outcomes Framework: what have you done to yoursleves? Br J GP, 2007: 435-7. 4. Marshall M, Harrison S. Its about more than money: financial incentives and internal motivation. Qual Saf Health Care, 2005: 14(1): 4- 5. 5. Paul C, Nicholls R, Priest P, McGhee R. Making policy decisions about population screening for breast cancer: the role of citizens' deliberation. Health Policy, 2007 (in press). Competing interests: Practicing GP whose practice receieves QOF funds
Re: It is the lack of evidence of the NET benefit of the framework itself, not the evidence around individual indicators that is the issue 29 November 2007
L Sam Lewis, GP Surgery, Newport, Pembrokeshire SA42 0TJ Send response to journal: Re: Re: It is the lack of evidence of the NET benefit of the framework itself, not the evidence around individual indicators that is the issue	So, Les Toop, everyone who agrees with Heath et al. is wise and well- meaning - and those who disagree are a bunch of self-righteous misguided folk. I don't think so. It took us 15 years to develop PRP form the initial DoH murmurings of a 'Good Practice Allowance' - and we have learned a great deal. Your view seems to be that we abndon any attempt at Performance-related pay, and return to the status quo ante ? Our negotiators WERE mandated to negotiate for it. We voted for it. We are now approaching the review stage - 1: should we scale it up or down in Financial value ? 2: should we alter any parameters ? I am convinced we should abolish the iniquitous square-root correct, precisely in order to push more resources to high-prevalence, high deprivation populations.. And I agree we should measure and minimise any harms ( what harms ?? ) and most importantly insist that the VOLUNTARY nature, and exception code facility be retained. Lets' take a BMA vote of British GP opinion on all elements of contract renegotiation , rather than rely on your gut feeling, straw poll, or convictions. Competing interests: 20% of practice resources come via QOF
Measure outcomes routinely 30 November 2007
Tim Benson, Director Abies Ltd, 14 Pinewood Crescent, Hermitage, RG 18 9WL, UK, Justin Whatling Send response to journal: Re: Measure outcomes routinely	Heath et al point out that the quality and outcomes framework needs to include clinically important outcome measures, not only measures of process, which are likely to improve outcomes. Clinical interventions are complex to evaluate but, despite this, there is generally only one endpoint of interest and that is improvement in the health and wellbeing of the patient. This single outcome can be measured through standardised health-related quality of life measures. We would argue that we should focus on measuring what matters to patients, that is, change in each patient’s health status over time. The requirement to routinely collect health outcomes data has been known since Florence Nightingale's time (1), yet this is still not done routinely. There are isolated examples such as the BUPA SF-36 programme monitoring surgery outcomes, but these are unusual. Recently there have been calls for outcome data to be collected as part of day-to-day care (2, 3). We need to have both cost and outcome data over the care cycle in order to progress the cost-effectiveness of care and promote value-based competition amongst providers. We believe progress has been slow in this area because all of the critical success factors of an outcomes measure have yet to be achieved: (a) Benefits to all stakeholders; (b) Takes almost no time to complete; (c) Can be used routinely (every encounter); (d) Technology platform- independent; (e) Contributes to patient care at the front line; (f) Applicable to majority of patients; (g) Reliable; (h) Valid; (i) Scalable. A tool called QoLive addresses these requirements and is currently in development. Michael Porter recognises that 'mandatory measurement and reporting of results is perhaps the single most important step in reforming the health care system' (4). So why don’t we do it? (1) Nightingale F. Notes on matters affecting the health, efficiency, and hospital administration of the British army. London: Harrison and Sons, 1858 (2) Wanless D. Our Future Health Secured? A review of NHS funding and performance. London: Kings Fund, September 2007 (3) Darzi A. Our NHS Our Future: Interim Report. London: Department of Health, October 2007 (4) Porter M and Teisberg E O. Redefining Health Care: Creating Value -based Competition on Results. Harvard Business School Press, 2006 Competing interests: The authors are developing the QoLife tool for measuring health status
Goodhart's Law 1 December 2007
Peter G Davies, GP Principal Keighley Road Surgery, Illingworth, Halifax. HX2 9LL Send response to journal: Re: Goodhart's Law	I welcome this paper by Heath et al. (1) I was also surprisingly underwhelmed by the somewhat pedestrian follow up paper from Lester and Roland (2) QOF is basically a deal with the devil. The devil here is the spreadsheet tyranny that characterises modern scientific Taylorism. It's an approach that says that what can be measured matters and that what is not measurable does not exist. It is allied to the false assumption that money is the sole motivation for quality work. (3) We currently have a central government culture of targets and imposed advice on performance across many areas of government activity. Despite this the government is missing the mark in most of the areas where it has set targets. I want to remind readers of Goodhart’s Law. (4) This states that when a measure (whether process or outcome) becomes a target it ceases to be a good measure. I think QOF will fall victim to Goodhart’s Law. Rotating measurements does not really avoid the problem as GPs will get round it by keeping registers up to date in the appropriate sequence to meet the targets. Persistently moving targets is the hallmark of bad and inconsistent management. If it is valid to reduce BP below 150/90 this year then it is next year as well. Heath et al’s point that QOF will favour practices in affluent areas where targets are easier to reach is well made. QOF could perversely become a mechanism driving doctors to move away from poor areas of cities with their more complex, and sicker patients with multiple problems to work in the better suburbs. It could end up rewarding those doctors who hit their targets more easily than those who struggle valiantly in our deprived areas. There will be many unexpected outcomes from QOF, and as the Chinese emperor, and indeed Marshall and Roland previously (5) commented, it may currently be a bit early to tell what they will be. References 1. Heath, I., Hippisley-Cox, J., and Smeeth, L. (2007) Measuring performance and missing the point. BMJ 335: 1075-1076 2. Lester, H. and Roland, M. (2007) Future of Quality Measurement BMJ 335: 1130-1131 3. Kay, J. (2007) The Failure of Market Failure Prospect magazine August 2007 36-42 4. http://economia.is-there.net/goodharts-law-controlling-is-distorting . (accessed 30.11.07) 5. Marshall M.; Roland M. The new contract: renaissance or requiem for general practice? British Journal of General Practice, Volume 52, Number 480, 1 July 2002 , pp. 531-532(2) Competing interests: NHS GP who is paid for work done for QOF. Voted against the New Contract in 2003 on grounds that it did not answer the questions about the meaning and purpose of GP work.
Goodhart's Law 3 December 2007
Richard Bartley, Physiotherapist Wales Send response to journal: Re: Goodhart's Law	The same reference used by Davies suggests there is a Reverse Goodhart’s Law. It implies that the implementation of a flawed system will eventually restore the status quo. For example, if the government’s credibility is undermined by its own incompetence then its targets become irrelevant and GPs will resort to more traditional methods of practice. Competing interests: A patient.
Dealing with conflicts between the QOF and patient care 4 December 2007
Graham N Box, Chief Executive National Association for Patient Participation Send response to journal: Re: Dealing with conflicts between the QOF and patient care	This is a fascinating and important debate. I would like to suggest the following from a lay perspective: 1. Quality frameworks do not inevitably undermine professionalism. The current arrangement whereby an important, but not overwhelming, proportion of practice income is Quality Outcomes Framework (QOF) related should see some of the poorest practices improved, while still allowing GPs the flexibility to provide person-centred care (however that is understood). 2. The exception reporting process is available for GPs who feel that QOF targets are not clinically indicated for particular patients, such as older people with hypertension. These exception reports should then feed into the process of challenge leading to future iterations of the QOF. It would be shameful if GPs are mistreating patients in order to secure QOF points. 3. We do need to develop far more sophisticated methods of rewarding practices who respond with greatest success to the needs of their patients and their communities. It is my view that this requires a stronger public health presence within primary care and a more effective community voice in commissioning. Can these be captured within the QOF? If not, what other mechanisms might be used to reinforce the relationship between practices and the communities that they serve? Competing interests: None declared