Re: When financial incentives do more good than harm: a checklist
Title : Evidence-based savings or financial alchemy?
Dear Sir/Madam,
Our policy makers would do well to read the excellent article on "Pay for Performance" by Glasziou et al as an object lesson in how to analyse the potential costs of change before committing to targets. (1) Two recently published policy documents, the Power of Information (2) and Digital by Default: the delivery choice for England’s population (3) have made extravagant claims for savings to be made by adopting apparently mundane IT solutions in the NHS. In the former total savings of £2.865bn have been forecast from implementation of a revitalised NHS Information Strategy of which £2.430bn arises from enabling direct patient access to medical records. In the latter, £2.9bn savings are proposed from 10 ‘easy-wins’ including £1.1bn savings from sending ‘negative test results’ to patients by SMS messages. All of these figures can be challenged.
Aside from potential double counting, digging behind these headline figures is revealing. The Power of Information is supported by a series of financial spreadsheets (4) which expose the ‘evidence’ behind the claims. The £2.43bn saving is achieved with an investment of just £152m, a return on investment that makes even banking profitability look modest. The total is marked as cash-releasing with 82% of the saving being in GP and nursing time. There is no mention of how the possible reduction 16,000 nursing or 11,000 GP WTE over 5 years would be delivered if real cash were to be saved (based on notional salaries of £100k and £50k for GPs and nurses respectively). Going deeper into the data reveals that all the figures are based on a draft of unpublished pilot studies of the impact of allowing patient access to a total of 189 patients which have been modelled to arrive at figures for a population of 46 million (5).
The second forecast is equally flawed. It suggests that there are 500 million tests per annum carried out in general practice and assumes that if an SMS message were sent informing patients of the 10% which are negative then the reduction in practice visits and follow ups would make the savings. There is no explanation of how the SMS messages would be generated, by whom nor of how the mobile phone numbers would be captured, stored and validated. However the most embarrassing flaw in the argument is that number of tests is likely to be highly exaggerated and based on a misreading of data from a recent safety audit of the Pathology Message Implementation Project through which laboratory tests are reported to GPs (6). Estimates from this work indicated that approximately 500m tests are reported each year but are sent in 37m test reports with, on average, 13.5 tests included in each. This is because tests are conducted as profiles such as full blood counts which constitute a functional test. Hence their figures are out by a magnificent 10 fold, or by 1000% in the new Osborne unit (7). The data as it stands fail any common sense check. Anyone working in pathology, either in the NHS or DH, would recognise instantly that £1.1bn is 50% of the total budget for all testing for both primary and acute care and that these figures smack of alchemy rather than science. A 50m reduction in appointments would equate to approximately 6,000 fewer appointments per GP practice per year or a 16% reduction on all primary care consultations (303 million per year in 2008-2009 (8)). To ascribe such reductions to negative testing is also fanciful when on average each GP practice sends in just 4625 test requests per year. One wonders if analysis of the other nine ‘easy-wins’ in the Digital by Default document would also reveal comparable flaws in their logic (3).
But the serious reason to question these figures is to consider how such nonsense can find its way into HM Government publications and can have passed through the scrutiny of the Cabinet Office and Treasury. Digital by Design boasts a stellar cast of academic advisors but has been produced by consultants with an upmarket address in central London. One can only assume that the academic advisors have failed to impose rigorous standards of scrutiny to the document or have been side-lined. If the latter is true then one would expect them to wish to distance themselves from the publication for fear of reputational damage. The major lesson should be that relying on paid advisors to write policy whilst ignoring the talents of highly capable civil servants or health professionals is dangerous. For conspiracy theorists this analysis adds evidence to the investigative journalism of Craig and Brooks of a progressive erosion of academia and the civil service by consultancy businesses (9).
Yours etc.
Dr Richard Jones
Yorkshire Centre for Health Informatics
University of Leeds
Leeds LS2 9JT r.g.jones@leeds.ac.uk
1. Glasziou P., et al. When financial incentives do more good than harm: a checklist. BMJ 2012;345:e5047 (Published 14 August 2012)
2. The Power of Information: Putting all of us in control of the health and care information we need. Department of Health, London, 2012
3. Digital by default The delivery choice for England’s population. Transform, Great Portland Street, London, 2012.
4. The Power of Information: Impact Assessment spreadsheet. (http://www.dh.gov.uk/prod_consum_dh/groups/dh_digitalassets/@dh/@en/documents/digitalasset/dh_134733.xls - accessed 15th August 2012)
5. Quote from speadsheet : cell A3 “Note: the assumptions for this action predominantly come from an early draft of a pilot study: Fitton, Fisher, Hannan et al "Examining the business case for Electronic Health Records Access by patients in two English General Practices".
6. See www.ychi.leeds.ac.uk/pmipunits (accessed 15th August 2012)
7. George Osborne's 110% – do the maths please, chancellor. The chancellor's consistent misunderstanding of percentages does not bode well for the economy. The Guardian London, Tuesday, August 7th, 2012.
8. Trends in Consultation Rates in General Practice 1995/1996 to 2008/2009: Analysis of the QResearch® database, Final Report to the NHS Information Centre and Department of Health, The NHS Information Centre for health and social care, September 2009. .http://www.ic.nhs.uk/webfiles/publications/gp/Trends_in_Consultation_Rat...
9. Craig D, Brooks R. Plundering the Public Sector. Constable and Robinson, London, 2006.
Rapid Response:
Re: When financial incentives do more good than harm: a checklist
Title : Evidence-based savings or financial alchemy?
Dear Sir/Madam,
Our policy makers would do well to read the excellent article on "Pay for Performance" by Glasziou et al as an object lesson in how to analyse the potential costs of change before committing to targets. (1) Two recently published policy documents, the Power of Information (2) and Digital by Default: the delivery choice for England’s population (3) have made extravagant claims for savings to be made by adopting apparently mundane IT solutions in the NHS. In the former total savings of £2.865bn have been forecast from implementation of a revitalised NHS Information Strategy of which £2.430bn arises from enabling direct patient access to medical records. In the latter, £2.9bn savings are proposed from 10 ‘easy-wins’ including £1.1bn savings from sending ‘negative test results’ to patients by SMS messages. All of these figures can be challenged.
Aside from potential double counting, digging behind these headline figures is revealing. The Power of Information is supported by a series of financial spreadsheets (4) which expose the ‘evidence’ behind the claims. The £2.43bn saving is achieved with an investment of just £152m, a return on investment that makes even banking profitability look modest. The total is marked as cash-releasing with 82% of the saving being in GP and nursing time. There is no mention of how the possible reduction 16,000 nursing or 11,000 GP WTE over 5 years would be delivered if real cash were to be saved (based on notional salaries of £100k and £50k for GPs and nurses respectively). Going deeper into the data reveals that all the figures are based on a draft of unpublished pilot studies of the impact of allowing patient access to a total of 189 patients which have been modelled to arrive at figures for a population of 46 million (5).
The second forecast is equally flawed. It suggests that there are 500 million tests per annum carried out in general practice and assumes that if an SMS message were sent informing patients of the 10% which are negative then the reduction in practice visits and follow ups would make the savings. There is no explanation of how the SMS messages would be generated, by whom nor of how the mobile phone numbers would be captured, stored and validated. However the most embarrassing flaw in the argument is that number of tests is likely to be highly exaggerated and based on a misreading of data from a recent safety audit of the Pathology Message Implementation Project through which laboratory tests are reported to GPs (6). Estimates from this work indicated that approximately 500m tests are reported each year but are sent in 37m test reports with, on average, 13.5 tests included in each. This is because tests are conducted as profiles such as full blood counts which constitute a functional test. Hence their figures are out by a magnificent 10 fold, or by 1000% in the new Osborne unit (7). The data as it stands fail any common sense check. Anyone working in pathology, either in the NHS or DH, would recognise instantly that £1.1bn is 50% of the total budget for all testing for both primary and acute care and that these figures smack of alchemy rather than science. A 50m reduction in appointments would equate to approximately 6,000 fewer appointments per GP practice per year or a 16% reduction on all primary care consultations (303 million per year in 2008-2009 (8)). To ascribe such reductions to negative testing is also fanciful when on average each GP practice sends in just 4625 test requests per year. One wonders if analysis of the other nine ‘easy-wins’ in the Digital by Default document would also reveal comparable flaws in their logic (3).
But the serious reason to question these figures is to consider how such nonsense can find its way into HM Government publications and can have passed through the scrutiny of the Cabinet Office and Treasury. Digital by Design boasts a stellar cast of academic advisors but has been produced by consultants with an upmarket address in central London. One can only assume that the academic advisors have failed to impose rigorous standards of scrutiny to the document or have been side-lined. If the latter is true then one would expect them to wish to distance themselves from the publication for fear of reputational damage. The major lesson should be that relying on paid advisors to write policy whilst ignoring the talents of highly capable civil servants or health professionals is dangerous. For conspiracy theorists this analysis adds evidence to the investigative journalism of Craig and Brooks of a progressive erosion of academia and the civil service by consultancy businesses (9).
Yours etc.
Dr Richard Jones
Yorkshire Centre for Health Informatics
University of Leeds
Leeds LS2 9JT
r.g.jones@leeds.ac.uk
1. Glasziou P., et al. When financial incentives do more good than harm: a checklist. BMJ 2012;345:e5047 (Published 14 August 2012)
2. The Power of Information: Putting all of us in control of the health and care information we need. Department of Health, London, 2012
3. Digital by default The delivery choice for England’s population. Transform, Great Portland Street, London, 2012.
4. The Power of Information: Impact Assessment spreadsheet. (http://www.dh.gov.uk/prod_consum_dh/groups/dh_digitalassets/@dh/@en/documents/digitalasset/dh_134733.xls - accessed 15th August 2012)
5. Quote from speadsheet : cell A3 “Note: the assumptions for this action predominantly come from an early draft of a pilot study: Fitton, Fisher, Hannan et al "Examining the business case for Electronic Health Records Access by patients in two English General Practices".
6. See www.ychi.leeds.ac.uk/pmipunits (accessed 15th August 2012)
7. George Osborne's 110% – do the maths please, chancellor. The chancellor's consistent misunderstanding of percentages does not bode well for the economy. The Guardian London, Tuesday, August 7th, 2012.
8. Trends in Consultation Rates in General Practice 1995/1996 to 2008/2009: Analysis of the QResearch® database, Final Report to the NHS Information Centre and Department of Health, The NHS Information Centre for health and social care, September 2009. .http://www.ic.nhs.uk/webfiles/publications/gp/Trends_in_Consultation_Rat...
9. Craig D, Brooks R. Plundering the Public Sector. Constable and Robinson, London, 2006.
Competing interests: No competing interests