Feature NHS reforms

How do we evaluate the impact of the health reforms on the NHS?

BMJ 2012; 345 doi: http://dx.doi.org/10.1136/bmj.e5539 (Published 21 August 2012) Cite this as: BMJ 2012;345:e5539
  1. Zosia Kmietowicz
  1. 1London

The Health and Social Care Bill came into force in March after much controversy, but how will we know if it’s good for public health? Zosia Kmietowicz asks doctors, health economists, and policy advisers

Even before it was put before MPs in January 2011 the Health and Social Care Bill was steeped in controversy for the monumental structural changes it would impose on the English NHS. By the time it received royal assent in March 2012 the 473 pages of legislation had been subjected to the most parliamentary scrutiny for nearly 20 years.

On the day it became law health secretary Andrew Lansley said the act would devolve power to doctors, give more choice to patients over how they are treated, drive up quality, enable integration, and reduce bureaucracy in the NHS. Local democratic involvement and public health would also be strengthened.

But how will we know if the act is good for the public’s health? We asked a range of doctors, health economists, and policy advisers what measures we could usefully look at to judge the impact of what Mike Farrar, chief executive NHS Confederation, described as “among the toughest projects the NHS has ever taken on.”1

Nick Black

professor of health services research, London School of Hygiene and Tropical Medicine, and chair, Department of Health National Advisory Group for Clinical Audit and Enquiries, London

It’s unlikely we will be able to determine the impact of any healthcare reforms for at least a few years. We only have to look back at GP fundholding to see an example of a reform that wasn’t given long enough for clear evidence of its impact to be rigorously determined.

Most of the government’s aspirations are motherhood and apple pie; what’s not to like about “ensuring a focus on integration” or “strengthening public health?” While these goals are also notoriously difficult to measure objectively there are a couple of aspirations that we could look at dispassionately: “drive up quality” and “give patients more information and choice.” The latter is already well under way having started under the previous government with NHS Choices and other initiatives. Many of the national clinical audits provide comparisons of providers’ performance and others are joining them by adopting a policy of public disclosure.

The principal challenge facing the government will be to demonstrate that the quality of care (safety, effectiveness, experience) has improved because quality in the NHS in England has been rising rapidly over the past decade.2 In a period of, at best, steady funding levels over the coming decade the NHS will be doing well to maintain recent improvements. The good news is that over the past decade national clinical databases and patient experience surveys have been established that will be able to determine what happens to some key aspects of healthcare.

John Quin

consultant physician, Brighton and Sussex University Hospitals NHS Trust, Brighton

I’ve just finished a Neo-Dickensian/Hogarthian ward round featuring various obese Pickwickian types, brutalised neglected Nancys, and some sodden Gin Lane funsters and the mere possibility of the health act showing signs of success in this double dip recession when there is no money left appears rather over-optimistic.

I’m reliably informed that in two or three years time there will be one of those general election things at which time modern vox pop sampling technology will inform us on a daily, no, near hourly, basis who favours what. And a quick glance at the percentages will be enough to draw implications about the success or failure of the health act. But here are some practical improvements that would be nice to contemplate as measures of, ahem, “success” as regards healthcare in England:

  • No more scorbutic or rachitic patients on ward rounds

  • No patients leaving wards to live in tents as opposed to proper housing

  • No patients waiting aeons for sheltered housing

  • Comedians and pop stars forced to repay tax that can help fund healthcare, and as an example perhaps be asked to help clean incontinent older patients. For a month. And that’s the doubly incontinent.

  • Moving down the Laziest Country in Europe rankings (currently third).Then we can all hop like a frog and go out on the street and do anything. At last we will be back in business and we will be . . . a success.

Peter Davies

consultant chest physician, Liverpool Heart and Chest Hospital, Liverpool

“How can I cure my patients unless I can control them,” is the cry of Zossimov, the doctor in Fyodor Dostoyevsky’s book Crime and Punishment and I have much sympathy with his predicament, both as a doctor and patient.

When it became apparent that I needed to start an anti-hypertensive drug I was grateful that my GP told me which to start on rather than offer me a bewildering choice of perhaps dozens. Yet when the local PCT [primary care trust] wanted me to change to a cheaper alternative I was round to my GP’s in a flash having seen too many people develop an intractable cough on the cheaper alternative on offer.

I would like to see a way of assessing whether there really is patient satisfaction in being offered a choice. I would like to see that assessment made now and again in three to four years’ time when the new bill will have had a chance to work through any changes in the doctor-patient power balance.

It is important because as things stand at present, patients can be left in some uncertainty as to the best choice to make and fear that their doctor is putting a choice to them because he or she may not be fully informed of the relative merits of the choices they offer. As one specialist registrar with a complex medical problem, who had good expert knowledge of his problem and who was offered a choice of treatment, pleaded with me, “I just want to be managed.”

Anna Dixon

director of policy, The King’s Fund, London

The challenge in evaluating the impact of health reforms is relating the decisions and actions taken by governments to the performance of the health system. This is challenging because reforms have multiple objectives, are emergent, overlaid on previous changes, their implementation varies depending on context and takes time, and political controversy can often colour interpretation.

The current reforms will be difficult to evaluate but this does not mean we shouldn’t try. As we said in our earlier response to the white paper “the same set of proposals could lead to very different end points depending on the process of implementation and the actions and reactions of those working in the NHS.” This suggests that any research evaluation will have to be flexible and iterative tracking the reform process in real time in a variety of different contexts.

Piloting and phased implementation enable more robust evaluation to take place. Devolution and the consequent divergence of healthcare systems between the four countries of the UK also provides a natural experiment for comparing the impact of different reforms. Currently the lack of comparative data and standardised measurement hampers such efforts.

The current government should move quickly to fund a reform evaluation programme that can track the impact of these reforms so that when the next politically contentious debate starts up it will at least be a little more informed.

Clare Gerada

chair, Royal College of General Practitioners, London

During the Health and Social Care debate I made a comparison between what will happen to our health service and what has happened to our rail service. Competition in the railway, far from reducing costs and improving quality, has resulted in fragmentation, increased costs, and reduced services—especially in those areas not deemed profitable enough.

General practice is already being eroded and shows early signs of fragmentation and reduction in services. From the citizen and patient perspective a key concern will be loss of data and ability to measure population access and outcomes within and between groups.

The ability to monitor and provide coordinated care to patients is essential. Much of the act is contradictory—on the one hand placing competition, choice and multiproviders as corner stones, but then emphasising the need for coordination of care, integration, and joint working.

Another area to monitor is the cost of the reforms. These are already mounting. For example, the costs of removing expensive and scarce GPs from the consulting room to serve the commissioning agenda will begin to show in reduced access to GPs, reduced continuity, and increased overall NHS costs.

There is a concern too that there will be several health services each with differing entitlements and new charges for services once free at the point of delivery. As with social care, we need to know these will be monitored.

Finally it would be important to monitor distribution of fair resources, staffing, and health inequalities, but the blurring of boundaries between public and private will make this very difficult.

Des Spence

GP, Glasgow

This seems a simple question but it is not. “Success” will largely depend on the parameters the government elects to measure. The temptation will be to choose highly selective measures and to look at crude activity data to “prove” the reforms a success. But quantity is no measure of quality of medical care where the best interventions are often no intervention. In addition, the introduction of assessments invariably leads organisations to indulge in gaming and manipulating the numbers rendering many assessments valueless.

Perhaps there are some useful areas to assess: prescribing targets, access to primary and secondary care, and patient and staff satisfaction. One outcome I hope will be scrutinised in detail is the profit sheets of the companies operating new NHS contracts.

But perhaps all my cynicism of assessments is a little premature, for of course this is a controlled medical experiment. These buccaneering free market reforms are only in England. In time we will be able to make direct, independent, and valuable comparisons between England and Scotland, Wales, and Northern Ireland. In reality it will be a decade before we can see the impact of these changes. Any government press release should come with a health credibility warning. Lastly, good healthcare is simple but unquantifiable: committed staff, a culture of care, and most of all, no profit for intervention.

Lindsey Davies

president, Faculty of Public Health, London

The transfer of public health to local authorities brings opportunities for a more integrated approach to improving and protecting people’s health. However, there are concerns that the act will fragment health and care services rather than strengthen them. The act might aim to give patients access to a wider range of care providers, but tracking the outcome will pose challenges.

The Faculty of Public Health (FPH) produced a risk assessment of the act, before it became law, with four main areas of concern: the loss of a comprehensive NHS, increased costs, reduced quality of care, and widening health inequalities. The measures, which could track whether these risks become reality, are largely to be found in the work of health and wellbeing boards, their joint strategic needs assessments, and the annual reports of directors of public health.

Changes in health, wellbeing, and inequalities can take years to effect and demonstrate. Short term, proxy indicators are required. Relatively simple measures, like tracking the size of the ring fenced public health budget and the number of directors of public health and public health consultants and specialists in post in England, will give an indication of whether good public health services are being delivered.

Allyson Pollock

professor of public health research and policy at Queen Mary, University of London

The changes being wrought by the health act are inconsistent with planned universal, comprehensive healthcare and the expected effects are growing differences in access to care, new charges for services, and an expansion of corporate control over clinical and management decision making.

The effects will be difficult to monitor. Comprehensive data currently routinely generated through the system of public administration will be lost under the fiendishly complex new system. Private providers will not have the same reporting duties as the NHS and their financial arrangements will be hidden under a cloak of commercial confidentiality. It will become increasingly difficult to gather systematic data about people’s entitlements to care let alone data about the care they receive.

In these circumstances the measures of success should not be outcomes, quality, and satisfaction ratings, each notoriously difficult to define. Instead, the first measures of success should focus on the public availability of information about equity of funding, patient and service entitlements, equity of access, population needs and planning data.

In short, the measures we need are those that will tell us the extent to which we have retained a universal comprehensive healthcare service. For example, what changes in access to services and entitlement to services have occurred? What services does each provider contribute to the NHS and under what commercial terms?

David J Hunter

professor of health policy and management, Durham University, Durham

I doubt that the changes made to the NHS will achieve much, if anything, that is positive, therefore trying to identify measures of success seems futile. The problems are threefold. Firstly, structural changes of this nature will take a long time to settle and require a recovery time of many years—certainly way beyond the next election; yet, the government is seeking quick results and evidence of improvement over the next couple of years which, in my view, simply can’t be done in a convincing way. Secondly, the fiscal context facing the NHS and local government—rather than the health act—is far more likely to be instrumental in determining any improvement (or not) in quality or public health. Thirdly, the act is likely to result in many of Lansley’s ambitions for the NHS going into reverse. For example, the reforms will result in more bureaucracy and not less; integration is likely to become more difficult and give rise to disintegration and fragmentation; and while public health going to local government may be the right move it has high risks, especially in terms of improving health and wellbeing at a time of general fiscal constraint.

I’m hugely sceptical about choice and its perceived significance. Most people do not want to have choice in healthcare—they want to be assured that local services are safe, accessible, and driven by professional values and not market style ones.

Whatever happens the government will be hell bent on claiming success for its changes if only to save face. They will most likely seek to massage, manipulate, and twist the evidence to suit their purposes regardless of what it might be telling them.

Jennifer Dixon

director, Nuffield Trust, London

The more pertinent question is what measures would be useful for tracking progress of the NHS given the highly challenging financial environment it faces over the next five years. Next to the economic challenge, the impact of the health act will be small.

The collection of measures in the NHS Outcomes Framework will be a good start. But I would like to see a range of metrics focusing on the quality of primary care. This area is relatively unscrutinised, yet pivotal if the NHS is to reduce avoidable hospital use. With the GP Extraction Service becoming available next year a vast amount of information will be available to analyse.

Information on cohorts of individuals over time would also be useful. It is possible to link person level data on use of inpatient, outpatient, accident and emergency, and shortly GP care in individuals in a way that no other country can. We should use this to track the pathways of care of cohorts of individuals to assess whether they may be receiving optimal care, and look at variations. In particular the care of older people who might be avoidably bouncing to and fro across a range of providers and social services.

Monitoring what is happening to the quality of care would also be valuable.

Nigel Edwards

senior fellow, The King’s Fund, Leadership Develop and Policy, London

Measuring health system performance is a poorly developed science. Being able to attribute changes to such a wide ranging set of reforms is even harder because the changes have created major disruption.

We should try as much as possible to specify what improvements we expect or the risk is, as always, the government will seize on an indicator that is moving in the right direction and claim that this is as a result of the reforms.

The reforms should produce much more clinical input into the design of care. If so there should be reductions in admissions for conditions where better community management is known to have an impact. Commissioning groups are beginning to turn their attention to the performance of primary care which will be commissioned nationally. So reductions in the variations in care, in particular in referral, standardised treatment rates, use of accident and emergency departments, and in access to primary care might be expected.

The reforms promise increased patient choice. The extent to which patients remember being offered choice and how far they exercise it would indicate whether the intention of the reforms was being met. Mapping these against changes in patient reported outcomes would give some early clues about how far the policy was achieving its purpose.

Another leading indicator for the success of the reforms would be a rolling survey of clinician attitudes including their ability to change the system they work in.


Cite this as: BMJ 2012;345:e5539


  • Competing interests: None declared.

  • Provenance and peer review: Commissioned; not externally peer reviewed.