Analysis

Collecting data on patient experience is not enough: they must be used to improve care

BMJ 2014; 348 doi: http://dx.doi.org/10.1136/bmj.g2225 (Published 27 March 2014) Cite this as: BMJ 2014;348:g2225
  1. Angela Coulter, associate professor 1,
  2. Louise Locock, director of applied research2,
  3. Sue Ziebland, professor of medical sociology2,
  4. Joe Calabrese, lecturer in medical anthropology3
  1. 1Nuffield Department of Population Health, University of Oxford, Oxford, UK
  2. 2Department of Primary Care Health Sciences, University of Oxford
  3. 3Department of Anthropology, University College London, London, UK
  1. Correspondence to: A Coulter angela.coulter{at}dph.ox.ac.uk

The NHS has been collecting data on patients’ experience of care for over 10 years but few providers are systematically using the information to improve services. Angela Coulter and colleagues argue that a national institute of “user” experience should be set up to draw the data together, determine how to interpret the results, and put them into practice

Anyone who has stayed in a hotel, flown in a plane, or bought something online will know that you are likely to be chased for feedback on the experience, usually during or shortly after the event. The NHS has caught the “real time” feedback bug too. By April 2015 all NHS patients attending any type of healthcare facility in England will be invited to report back on their experiences using a variant of the friends and family test (FFT)—how likely are you to recommend our service to friends and family if they needed similar care or treatment? Real time feedback, often gathered on-site using handheld electronic devices, has been introduced into many hospitals over the past few years, and since 2013 it has been mandatory to offer all patients in acute hospitals and maternity services an opportunity to complete the FFT.1 In December 2014 the requirement will be extended to general practice, community, and mental health patients and then to all remaining services over the following three months. This survey is just the latest in a series of government led initiatives to measure patients’ experience (box 1). However, less effort has gone into how to understand and use the data, and there is little evidence that the information has led to improvements in the quality of healthcare. We suggest how the data could be used more effectively.

Box 1: Regular national health and social care surveys carried out in England1

Abundance of data

People’s experiences can be studied in a a wide range of ways, encompassing both narrative studies and surveys.2 Regular national or regional surveys of patients’ experience of care have been introduced in most of Europe, Australia, Canada, Hong Kong, Japan, South Korea, Mexico, New Zealand, and the United States.3 England was a pioneer, being the first country to introduce mandatory surveys of hospital inpatients, which have been carried out annually since 2002. The national survey programme has since been extended to encompass many other aspects of care (box 1). These surveys, which mainly use self completion postal questionnaires, cover topics such as access and waiting times, provision of information, communications with health or social care professionals, quality of the physical environment, involvement in decisions, support for self care, coordination of care, health status, and quality of life. Scotland’s patient survey programme, the Scottish Patient Experience Surveys, use different questionnaires, which means that the results cannot be directly compared with those from England.

As well as participating in national surveys, many health and social care organisations conduct local surveys using postal, telephone, face-to-face, or online methods. Meanwhile, the General Medical Council requires all doctors to gather patient feedback on their performance as part of the appraisal and revalidation process, and standardised questionnaires have been made available for the purpose.4

Qualitative (non-statistical) methods are also widely used by provider organisations. These include in-depth interviews, video interviews (for example, www.healthalkonline.org and www.patientvoices.org.uk), focus groups, patient forums, patient diaries, mystery shopping (participant observation), complaints and compliments, comment cards and suggestion boxes, informal feedback, and free text web based feedback (for example, www.patientopinion.org.uk, www.iwantgreatcare.org, www.nhs.uk).

Matching methods with needs

There is no perfect method for gathering experience data, so it is important to be aware of the strengths and weaknesses of the different approaches.5 Surveys generally use probability samples with the aim of obtaining representative results. Reliable interpretation of the data depends on having full information about the survey population, the sampling frame, the way in which the data were collected, the achieved sample, and the response rate, because imperfectly drawn samples and low response rates carry a higher risk of producing biased results. The FFT surveys are intended to be used with all patients, not just a sample, but response rates in the latest set of published data ranged from 15% for patients attending emergency departments to 29% for inpatients, with considerable variation between trusts. Organisations have freedom to gather the data in different ways. This variation may not matter if the data are to be used locally, but they cannot be used to make reliable comparisons between organisations or over time.

Simple evaluative questions like the FFT tend to produce more positive responses than more detailed questions asking about specific events, with only weak correlation between global scores and detailed reports.6 Response rates to postal surveys are generally falling, suggesting an element of survey fatigue—for example, responses to the Care Quality Commission’s national inpatient surveys dropped from 64% in 2002-03 to 51% in 2012-13 (www.nhssurveys.org). However, this has occurred at a time when unsolicited feedback through social media is burgeoning.

Narrative methods, such as in-depth interviews or focus groups, can produce richer, more detailed data than questionnaires with fixed response options. They are used to obtain accounts of people’s experiences and the way they explain or interpret these. Qualitative data are usually reported using words, not numbers, and it is harder to make comparisons or generalisations. Interviewing, transcribing, and analysing the data are skilled and time consuming tasks.

Stories trump data for capturing the interest of staff, and video interviews have been used to good effect to help staff view their services through patients’ eyes. Experience based codesign (EBCD) is a participatory action research approach to service improvement that draws on narrative interviews with patients about their experiences of care, as well as staff interviews and ethnographic observations.7 Patient interviews are video recorded and analysed for “touchpoints”—key moments of interaction between patient and care system where quality can be improved. A “trigger film” illustrating this analysis is shown to both patients and healthcare professionals, who then work together to implement agreed improvements. Local interviews have traditionally been used as the basis for EBCD, but recent research has shown that nationally collected video interviews can also be used effectively.8

There is also a great deal of interest in the use of Trip Advisor-style websites to gain feedback,9 especially now that 80% of UK households have internet access and all new mobile phones are internet enabled.10 However, the proportion of the population online drops to only 36% of single households where the householder is aged 65 and over, and older people are the heaviest users of healthcare, so there is a risk of bias from these methods too. Nevertheless, US hospitals collecting data by both web rating and traditional surveys found positive correlations between the two methods.11 They also noticed a correlation between both these types of feedback and other indicators of clinical quality (mortality and readmissions), adding support to evidence of a relation between patients’ experience and clinical outcomes.12 13 14 Patients who report good healthcare experiences tend to respond better to treatment, as judged by clinical indicators (such as recovery from myocardial infarction, blood glucose levels, and infections) and quality of life measures. However, a recent British study found no association between comments about hospitals on Twitter and their patient survey results.15

How are data being used?

We already know a considerable amount about the aspects of care that patients and families think are important, from routine surveys and a large body of quantitative and qualitative research.16 In the UK, the National Institute for Health and Care Excellence has published a quality standard on patients’ experience and guidance on best practice.17 While most NHS patients give mainly positive reports about their experiences, some problems crop up repeatedly: the failure to provide appropriate information about prognosis and treatment; not enough involvement in decisions; weak support for self care; lack of empathy and emotional support; fragmented and poorly coordinated services.

So has all the data collection led to improvements? At first sight the answer is somewhat depressing—clinicians often ignore survey evidence.18 19 For example, findings from the nationally organised GP patient survey often engender defensive reactions from the profession or lack of interest and are sometimes mistrusted.20 And after more than 10 years of gathering patient experience data, only a minority of hospital providers have been galvanised into taking effective action.21 A study of clinicians’ attitudes in four countries (Denmark, Israel, UK, and US) found “a chasm” between hospital managers’ assertion of the importance of patients’ experience and their failure to engage clinicians or provide structured plans for improving it.22 It seems that measurement is necessary but that change will not happen without effective leadership improvements (boxes 2 and 3).

Box 2: Use of survey data to improve communication with patients23

Stillwater Medical Group, a multispecialty group practice in Minnesota, participated in a state-wide pilot to measure and publicly report patient experience data. After examining the patient survey results, it decided to focus on an item that asked patients if they had received easy to understand instructions about self care.

This initiative coincided with the introduction of a new electronic medical record system that included a module for documenting and printing an after visit summary for the patient to take home. The module was embraced much more quickly in the internal medicine department (51%) than in the family medicine department (6%), allowing for a natural comparison.

Use of the module was seen to have a direct effect on patient survey scores, with positive responses for internal medicine rising from 84% to 98% of patients saying they definitely received easy to understand instructions. Meanwhile responses from family medicine patients languished at 72%. The discrepancy in survey scores and the demonstrable improvements achieved by the internal medicine department helped to galvanise action in other departments. This was achieved by a combination of effective physician leadership, regular quarterly measurement of patients’ experience, and the stimulus of internal competition.

Box 3: Using patients’ stories to improve the admission process24

Being admitted to a mental health ward can be distressing. Initial attempts by management and staff at Oxleas NHS Mental Health Trust to improve the admission process had produced disappointing results. They decided to try an experience based codesign approach, using videos of patients talking about their experiences as a trigger for discussion between staff and patients about the potential for change.

Using this approach in a mental health setting raised some unique challenges. Patients sometimes came face-to-face with a staff member around whom they had felt uncomfortable in the past, so they needed preparation, support, and self-control to focus on co-design rather than recriminations. Many worried about how speaking out might affect their experience if they were readmitted. For staff too, the experience was potentially exposing. Senior management played a vital role in allaying scepticism and emphasising that this was a design project rather than an investigation.

The theme that emerged from the videos was the need for more communication between staff and patients, especially during the admission process. The admission process was an extremely busy time for staff, but for patients the dominant experience was of waiting and feeling highly anxious. Seeing the process through patients’ eyes had a strong effect on staff, who decided to change their accelerated triage system to a model of care that was more aligned to service users’ needs. Feedback from daily groups on the ward, based on the emotional touchpoints identified during the discussions, was linked back to staff supervision and became incorporated into the primary tasks of the ward. The process led to an 80% drop in the number of complaints over the following year.

Research has begun to shed light on the factors that influence the likelihood of change (box 4).25 At the organisational level these may include the quality and commitment of the leadership, clarity of goals, identification of dedicated champions, active engagement of patients and families, staff skills, training and capacity, and availability of resources, in addition to the quality and depth of understanding of the patient’s perspective.26

Box 4: Essential elements of a strategic approach to improving patients’ experience 25

  • Strong, committed senior leadership: Success and sustainability depend on effective distributed leadership with active support from the chief executive and board. Support must be empowering rather than directive, enabling people on the front line to innovate without fear of retribution if things don’t turn out as expected

  • Dedicated champions: A dynamic champion whose central focus is to improve patients’ experience is essential for driving change at the operational level

  • Active engagement of patients and families or carers: Can range from patient involvement in organisational decisions (such as service developments) to engagement at the point of care

  • Clarity of goals: Clear goals and standards, plus effective methods for communicating these at every level, help spread and reinforce patient centred values and procedures

  • Focus on the workforce: Use of patient and carer feedback in staff development and appraisal can be helpful. Some organisations also include patients on interview panels. The staff culture and work environment should be constantly developed and reviewed

  • Building staff capacity: Staff require special training in communication skills and quality improvement concepts and methods. Also highlight examples of new roles, tools, and initiatives that have been shown to work well elsewhere

  • Adequate resourcing: Support is needed for systems that help improve care—for example, introducing new appointment and scheduling systems, improving access arrangements, or developing facilities for family members to stay overnight

  • Performance measurement and feedback: Continuous measurement and reporting on patients’ experiences to assess progress, strengthen accountability, and identify new opportunities for improving performance. Both narrative feedback and surveys have a role

Coordinated approach to improving care

A more coordinated approach is needed if we are to make better use of people’s reports on their experiences. This should bring together the various data sources, enabling more in-depth analysis of these, exploring linkages and overlaps, developing and testing more efficient ways to gather the data, and working out how to ensure that the results are used for quality improvement. The strong policy focus on measuring experience has not been matched by a concerted effort to develop the science that should underpin it, so improvement of the knowledge base about the factors that influence patients’ experiences will also be important.

The National Institute of Health Research has no funding stream dedicated to relevant methodological innovations in this field, and the large databases of national survey results and narrative accounts of patients’ experiences have been underexploited.21

Three national survey coordination centres are responsible for designing the questionnaires for the NHS patient experience surveys, overseeing their implementation, collating and publishing the results: CQC surveys, the general practice patient survey, and the national cancer patient experience survey. There is no comparable national focus for narrative studies, nor for supporting clinicians in making use of both quantitative and qualitative data for quality improvement. A national institute of patient (or service user) experience, drawing together the work of the survey coordination centres, academic groups with relevant interests, and specialists in implementation science, would give a much needed focus. It could span the methodological divide between quantitative and qualitative approaches, studying the most effective ways to combine them. It should have a largely practical focus, working with providers to experiment with different ways of gathering and using patient experience data at clinical, organisational, and policy levels to stimulate improvements. Associations between patients’ experience and staff experience, as collected by the NHS staff surveys, should be explored, and further work is needed to develop effective methods for studying experiences that span organisational boundaries.27

The Consumer Assessment of Health Plans initiative in the US may provide a useful model. Commissioned by the national Agency for Healthcare Research and Quality, with the work led by a group at Yale University, together with the Rand Corporation of California and other agencies, the programme has developed through three five-year phases. The first 10 years focused on the development of survey instruments and reporting formats, while the latest phase has emphasised use of the data for quality improvement.

Conclusion

People’s emotional and practical response to illness and the responsiveness of health providers and systems to their needs is crucial, both because it matters hugely to all users of healthcare and because it has a direct influence on the other dimensions of quality. Careful observation, measurement, recording, interpretation, and analysis of patients’ subjective experiences are essential to appreciating what is working well in healthcare, what needs to change, and how to go about making improvements. It is unethical to ask patients to comment on their experiences if these comments are going to be ignored. We believe a more concerted attempt is now required to make use of the evidence. An institutional focus could prove to be the key to getting it taken more seriously.

Notes

Cite this as: BMJ 2014;348:g2225

Footnotes

  • Contributors and sources: The article draws on a book that was jointly edited by the authors, Understanding and Using Health Experiences. All authors have carried out research into patients’ experiences. AC is the guarantor.

  • Competing interests: We have read and understood the BMJ Group policy on declaration of interests and have no relevant interests to declare.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

References