Reviving the NHS—lessons from Labour 1997-2005

We have a great deal to learn from the period 1997 to 2005 when, as the King’s Fund has reported, the English NHS began the greatest improvement in its history.1 I was directly involved as a regional director, and from 2000 as NHS chief executive and permanent secretary of the Department of Health. This short article doesn’t cover everything and is inevitably subjective. Others will have different perspectives and I hope that this article will provoke debate, discussion, and, most importantly, learning for the future.

At the end of the 90s the NHS was in a bad state, with long waiting lists, staff shortages, and public concern about nosocomial infection and hospital cleanliness. Outcomes were poor compared with similar countries, and many emergency departments were in chaos. UK expenditure on health was around 25% below that of comparator countries. There were some obvious similarities with today’s situation, although the current position is clearly even more serious and difficult to resolve.

Between 1997 and 2005, major investment and very substantial improvements were made in the NHS. The emergency department target of 98% of attendances being completed in under four hours was achieved in April 2005 (up from about 80%), surgical waiting lists fell by more than 40% to below 800 000 with a maximum wait of six months, and preventable mortality from cardiac causes and cancer also fell.23 Public confidence in the NHS doubled between 1999 and 2009, and by 2003 this was reflected in a substantial and continuing fall in the number of patients using the private sector for elective surgery.4

This was a period of enormous progress, but also failings. Progress was too slow in the first three years. Important developments occurred in policy on quality between 1997 and 2000 which had a profound long term impact, but saw no overall change of direction or improved performance. The National Institute for Clinical Excellence (NICE) was established to support decision making; along with the beginning of a system of National Service Frameworks which set standards in specific specialities; and the Commission for Health Improvement was created as the first national inspection and reporting system.

A new direction was established in July 2000 with the publication of the NHS Plan.5 Planning was only the start. It took about a year for the NHS to be mobilised around the new direction and for the continuing decline of the service to be reversed. Substantial improvements only began to appear in 2002—in Labour’s second term in government—by which time politicians and the public were very impatient. Improvement accelerated very fast from there.

Momentum for change and buy-in for the plans was created very successfully in 2000 and 2001 through three related developments—the creation of the NHS Plan, the prime minister’s public commitment to increase spending to the European average, and the Treasury’s commissioning of the Wanless report on long term sustainability.6

The NHS Plan set out a vision for the future which was supported by policy change and increased funding. Crucially, it was developed through a collaborative process involving people from all parts of the NHS and some of its partners, and the foreword was signed by representatives including the British Medical Association and the Royal College of Nursing. The Plan created enough goodwill to carry the government through some very difficult decisions in the next two or three years, for example about the use of the private sector.7

Political momentum was vital, but Labour, like all incoming governments, brought some ideological baggage into government. It chose very early on to change NHS consultants’ contracts to abolish their private practice, which was unfinished business from an earlier administration. This may have been a good thing in principle, but the timing was awful coming so soon after the publication of the NHS Plan, and meant that it lost some important allies and delayed progress. The issue was finally resolved pragmatically after the 2001 election, with changes to the contract agreed which tightened control of private practice but didn’t abolish it.

Progress was also hampered by political infighting between prime minister Tony Blair and health secretary Alan Milburn on one side, and the chancellor of the exchequer Gordon Brown, on the other. Each side had different visions for health: one broadly consumerist and health service orientated, the other more concerned with deeper causes and population health.

Progress was very fast once the new systems and approaches were established and supported by a strong performance management and accountability structure.

During this period, stronger management, targets, and improved accountability were being introduced across the whole of government. These enabled a clear focus, which was vital in making progress, but could cause damage when applied inappropriately. Some NHS targets were very useful. Early focus on improving cardiac waiting lists (which had 4% mortality) provided an early success. The emergency department target, which was placed on a whole hospital, not just the department, meant processes throughout the hospital had to change and specialities had to take more responsibility for their own patients from the moment they arrived. It enabled a more systematic approach to patient flow through the hospital.

Important advances also took place in management, with the Modernisation Agency as a vehicle for identifying and spreading best practice. Improvements in emergency department performance came from the use of systematic improvement processes alongside focused performance management. Other improvements in clinical practice came from new research and development strategies, the Cochrane Collaborations, NHS Evidence, and NICE.

There were, however, too many targets. The NHS plan had more than 200. Some were badly thought out, ineffective, and even counterproductive. Few were truly systemic or strategic. All of them led to some gaming and a common criticism of “hitting the target while missing the point.”

The focus was also too much on quantity and not enough on quality. The NHS Plan, for example, set a target of recruiting 20 000 more nurses, and almost 70 000 more were recruited by the end of 2005.8 However, the emphasis was on numbers recruited and not on speciality, job role, or location, or even on the quality of the workforce. Political needs dominated: a political party that had been in power since 1997 had to show measurable improvements. This led to missed opportunities and higher costs when organisations expanded too fast in response to political pressures.

Underlying these issues was the continuing problem of getting the right balance between national and local control and priorities. This led to far too many reorganisations and a regular tightening of central control when there were problems, and relaxation when things improved. This was demoralising and confusing.

Several very effective policy changes took place. Two of the most controversial were the introduction of patient choice over the location of their treatment, and use of the private sector. Others, such as the introduction of drop-in centres and minor injury units, were uncontroversial, while the introduction of prescribing by nurses and other professionals began controversially but, because of effective implementation, was soon completely accepted.

Improvements in surgical waiting times resulted partly from spreading good practice and tight management, but also initially from increased patient choice and the introduction of treatment centres operated by the independent sector. Interestingly, in both cases the greatest impact resulted not from patients exercising choice (very few did), or the small numbers being treated in these centres, but from NHS units responding to the competition and speeding up admissions. The threat of competition was very effective. Here, as elsewhere, the policy and politics were helpful, but their impact should not be overstated. The fastest improvements came in well run hospitals and surgeries where staff shared commitment to progress and their working relationships were good. Disruption of these relationships was one reason why frequent reorganisations were so destructive.

These changes were part of an approach to the NHS inspired by the private sector which created a quasi-market and led to some reorganisations. This undoubtedly brought some benefits. Professionals need to understand costs of treatments. Choice and competition can break up cosy cartels and help shift power to the patient. However, this economic centred approach failed to deal adequately with the human aspects. There is scope for the managed involvement of the private sector in NHS delivery, as now, but it needs to be on the NHS’s terms, and economic initiatives need to be part of a wider set of policies that address the realities of patients’ and professionals’ lives.

Two underlying problems persist. Firstly, much of healthcare needs to be long term and relational, not transactional and episodic, if it is to be successful. Chopping and changing providers in many cases damages care. Secondly, the emphasis on economic incentives rather than on human motivation doesn’t work in a service whose professionals are not motivated by money (although they can be demotivated by lack of it, as we can see from this year’s industrial action), but by achievement, respect, and vocation.

The development of management, systematic improvement processes, and improved accountability were a counterbalance to the power of the professions that had dominated the NHS, and were in decline. This decline was due in part to major scandals being revealed: about cardiac surgery in Bristol, the murders of his patients by Harold Shipman, and the unauthorised retention of children’s organs at Alder Hey. These changes, away from “a club culture” in some parts of the professions, were necessary, but were not balanced by the development of a clear vision of professionalism for the future. Looking ahead, these human aspects need to be prioritised if the NHS is to thrive.

There was a failure to pivot away from health services towards health and care. The service improvements meant we knew by 2004 that we had an opportunity to move upstream into developing community services (including social care), promoting health, and preventing disease. Various policy papers were published, but no political will drove this forward for a variety of reasons, including the perfectly rational desire not to give up on progress with services. The wise findings of the Wanless review, namely that sustainability requires the full engagement of the population, were ignored. The opportunity that had been created was lost.

We all have reason to regret this failure to change direction. The coalition government of 2010-2015 introduced austerity and destroyed some of the social fabric in our communities which, among other things, helped keep people healthy. This has in the longer term increased ill health and, together with the pandemic, damaged services.

My personal involvement in the NHS ended in 2006, not long after a new political team was appointed. We had devolved too fast without sufficient central controls and ran into financial problems. I resigned and the new chief executive imposed tighter central control. Progress on health services resumed, but there was no pivot to health.

An enormous amount has changed in 20 years, including public expectations and the wider social context, and I wouldn’t want to overplay the similarities between then and now. However, the basic problem of sustainability remains. People who propose new funding or organisational arrangements as solutions to today’s problems make a fundamental category mistake. This is not a financial or organisational issue but a health one. This means we need a health based solution that can then be followed through with financial, organisational, and other support.

I would draw out four key lessons:

Firstly, whatever government is in power after the next election should establish a new long term health plan that sets out a vision for the future that goes far beyond the health and care systems and backs it with policy change and financial support. Crucially, it should be developed through a collaborative process involving people from all parts of the health and care systems as well as wider society, which does so much to determine our health and wellbeing. This will provide both the direction and the momentum needed for implementation.

Secondly, the long anticipated shift of care from hospitals to the community and homes needs to be planned for, funded, and accelerated. This can be supported by advances in science, technology, and data, which will determine much of the framing and the language of health, shape how health workers think about health problems and possible solutions, and how they act.

Thirdly, we need to pivot to health even more than before. The pandemic has shown us all—even if we haven’t studied the research on the social determinants—that housing, the environment, communities, employment, poverty, and education profoundly affect our health, wellbeing, and life chances. We need health policy that tackles all this with new emphasis on creating health (providing the conditions for people to be healthy), preventing diseases, and protecting the health of the population alongside excellent health and care services.9

Finally, we need a new emphasis on people. Not just on numbers, but also on people’s motivations, their experience as patients and professionals, the changing roles we can anticipate, and the development of professional education.10 Ultimately, of course, it was health and care workers who achieved the NHS’s successes.