Intended for healthcare professionals

Analysis

Measuring patients’ experiences and outcomes

BMJ 2009; 339 doi: https://doi.org/10.1136/bmj.b2495 (Published 02 July 2009) Cite this as: BMJ 2009;339:b2495
  1. Nick Black, professor of health services research1,
  2. Crispin Jenkinson, professor of health services research2
  1. 1Department of Public Health and Policy, London School of Hygiene and Tropical Medicine, London WC1E 7HT
  2. 2Department of Public Health, University of Oxford, Oxford OX3 7LF
  1. Correspondence to: N Black nick.black{at}lshtm.ac.uk

    Increasing importance is being given to patients’ views of the humanity and effectiveness of their care. Nick Black and Crispin Jenkinson consider the challenge of using them to evaluate interventions and assess the quality of services

    There is now a widespread realisation that patients’ views are not optional but essential to achieving high quality care.1 The contribution of patients to NHS funded research in England is enshrined in the national advisory group INVOLVE,2 and the 2008-9 NHS operating framework took the first step in making patient reported outcomes a mandatory requirement for audit.3 In addition, national surveys of patients’ experiences of health care have become a feature of NHS regulation over the past few years.4

    Patients offer a complementary perspective to that of clinicians, providing unique information and insights into both the humanity of care (such as dignity and respect, privacy, meeting information needs, waiting and delays, and cleanliness of facilities) and the effectiveness of health care. Given that, apart from when they face life threatening events, patients are generally seeking alleviation of symptoms, reduced disability, and improved quality of life, it makes sense to involve patients as they, rather than clinicians, are the ones who can report on these aspects of their health.

    Greater recognition and acceptance of the contribution of patients over the past two decades has been accompanied and facilitated by a major growth in the development of rigorous instruments (questionnaires) and a burgeoning research literature on their uses and benefits. This article outlines some of the recent developments and initiatives in England before considering some of the methodological, practical, and political challenges.

    Patients’ experiences of humanity of care

    Over the past decade, concern has shifted from measuring patients’ satisfaction with the humanity of care to measuring their experience of the humanity of care (that is, seeking value free reports of cleanliness, dignity, etc). This is because the measures of satisfaction were strongly influenced by patients’ expectations, which could be quite low, and were found to be fairly insensitive in detecting shortcomings. However, the use of patients’ experiences means that other people, such as samples of the general population, decide whether the humanity of care was acceptable.

    Four yearly national surveys of patients’ experience of general practice were instituted in 1998, coordinated by the Healthcare Commission, followed by annual surveys of acute inpatients in 2001.4 In addition, occasional surveys of specific services, such as coronary heart disease and maternity services, have been carried out. The results are used by commissioners to track performance over time and inform improvement activities, by regulators in providers’ annual health checks, and by the Department of Health to measure progress against national objectives. Many providers and practitioners also carry out local surveys, often using locally produced questionnaires of uncertain reliability and validity.

    Box 1 shows examples of the types of questions included. In well designed instruments, the topics covered and the phrasing of questions are derived from preliminary qualitative inquiries. These include in-depth interviews with a sample of patients followed by pilot testing of draft questionnaires. Although qualitative approaches that produce narrative accounts are of limited value in the routine auditing of services, they can provide rich insights and may be useful to future patients and their carers.5

    Box 1 Examples of questions included in two patient reported experience questionnaires

    Picker Institute inpatient questionnaire
    • When you had important questions to ask the doctor, did you get answers you could understand?

    • Yes, always Yes, sometimes No I had no need to ask

    • Were you involved as much as you wanted to be in decisions about your care and treatment?

    • Yes, definitely Yes, to some extent No

    Picker Institute patient experience questionnaire
    • Did doctors talk in front of you as if you weren’t there?

    • Yes, often Yes, sometimes No

    • If you had any anxieties or fears about your condition or treatment, did a nurse discuss them with you?

    • Yes, completely Yes, somewhat No Didn’t have anxieties or fears

    Patients’ reports of outcome

    Although clinicians can make objective observations of signs, impairment, and disability, only patients can report on their symptoms and quality of life. Many instruments exist for reporting these aspects. Most are specific for a condition or procedure. For example, the Oxford hip score for patients with hip osteoarthrosis6 and the international prostate symptom score for patients with benign prostate disease.7 Box 2 gives examples of the type of questions included. Although referred to in the UK as patient reported outcome measures (PROMs), they are actually measures of health status; the outcome of care has to be derived by comparing data collected at different times (for example, before and after surgery). There are also some generic instruments (such as the EQ-5D and the SF-36) that are designed to be used with any patients. These are less sensitive than disease specific instruments but allow comparisons between patients with different conditions.

    Box 2 Examples of questions included in two patient reported outcome questionnaires

    Oxford hip score
    • During the past four weeks, how would you describe the pain you usually have from your hip?

    • None Very mild Mild Moderate Severe

    • During the past 4 weeks, have you had any trouble with washing and drying yourself (all over) because of your hip?

    • No trouble at all Very little trouble Moderate trouble Extreme difficulty Impossible to do

    Aberdeen varicose vein symptom severity questionnaire
    • Does the appearance of your varicose veins cause you concern?

      • No

      • Yes, their appearance causes me slight concern

      • Yes, their appearance causes me a great deal of concern

    • During the last two weeks, have your varicose veins interfered with your leisure activities (including sport, hobbies, and social life)?

      • No

      • Yes, my enjoyment has suffered to a slight extent

      • Yes, my enjoyment has suffered to a moderate extent

      • Yes, my veins have prevented me taking part in any leisure activities

    Directories8 and systematic reviews of available instruments exist to help select the most appropriate one. The better directories provide independent assessment of the validity, reliability, responsiveness, and feasibility (cost, time to complete, acceptability) of instruments. But that is not sufficient; instruments must be used rigorously to ensure confidence in the findings: are the patients representative? are sufficient numbers included? is there recruitment bias? are the outcomes measured at the appropriate time after an intervention?9 10

    Until recently, patient reported outcome measures have been used only in research. However, since April 2009, all NHS patients in England having hip and knee replacement, hernia repair, or varicose vein surgery have been invited to provide information on their health status before surgery and three or six months afterwards.3

    Challenges

    Current enthusiasm for involving patients in measuring the humanity and effectiveness of care should not obscure the challenges. Although the use of patient reported outcomes in clinical trials has been well documented,11 their application in routine quality assessment has received less attention. Three areas need to be considered: methodological challenges, practical considerations, and interpretation.

    Methodological challenges

    If meaningful comparisons of providers are going to be made then two methodological problems need to be addressed. Firstly, the sample sizes need to be sufficient to ensure differences are statistically meaningful. Decisions about the required level of certainty are political rather than technical. The answer depends on the importance attached to missing a poor performer compared with unfairly labelling a satisfactory provider as poor. If the former is more important, then providers lying more than 2 standard deviations from the mean need to be identified; if the latter, then greater than 3 standard deviations is appropriate (fig 1).

    Figure1

    Fig 1 Funnel plot of incidence of patient reported complications after hip replacement for 11 providers in England, 2006-7.8 Mean proportion was 32.5% (horizontal line); 2 standard deviations (dotted curves); 3 standard deviations (solid curves). Each provider is shown by an open dot (crude proportion) and a solid dot (case-mix adjusted proportions)

    Secondly, it is essential to take differences in case mix between providers into account. These may arise because the eligible patients differ or because low recruitment rates introduce a bias. In a recent survey of patients having day surgery (fig 2), less healthy people were less likely to participate, introducing a small but significant bias into comparisons of outcomes between providers because less healthy people were shown to be less likely to benefit from treatment.12

    Figure2

    Fig 2 General health status of patients (EQ-5D score) having day surgery for hernia, varicose veins, or cataracts by proportion of patients recruited (32 providers in England, 2006-7).8 The regression line indicates that an increase of 20% in the proportion of eligible people recruited is associated with a decrease in their mean general health status of 0.015 units in the EQ-5D score (Pearson correlation coefficient −0.38, P=0.03)

    Practical considerations

    The best time to request patients’ views is not always obvious. Generally speaking, views on the humanity of care need to be assessed fairly soon after using a service to prevent recall bias. However, a delayed response may help distinguish between minor irritations and serious failings. Some delay is needed when obtaining patients’ views of outcomes to ensure all potential benefits have been realised. However, if the delay is too long, patients may struggle to remember any complications they experienced soon after the intervention.

    It is more challenging to measure patients’ views of outcomes than their views of the humanity of care, particularly after emergency treatment and long term care. The effectiveness of elective surgery is relatively straightforward to assess, as patients have a discrete intervention and their health status can be assessed before and after, but long term conditions, such as diabetes, require assessments at regular intervals (perhaps six monthly) and the goal may be to maintain, or avoid deterioration, in health status rather than a significant improvement. Acute emergencies represent another challenge as it is impossible to obtain the pre-event health status of patients (except by recall, which is of uncertain validity). One option is to compare the patient’s report of their health status once they have recovered from the acute episode with age-sex standardised population norms.

    Patients who are unable to self report because of physical or mental impairment present another challenge. One option is to rely on their lay carers (relatives, friends) to act as proxy respondents. Clearly, the danger is that the reported views partly depend on the proxy’s observation and interpretation. An alternative approach is to seek the views of the lay carers rather than expect them to second guess the patient’s views.

    Interpretation

    Interpretation of the scores obtained from instruments presents some challenges. Before a survey is done the measurement properties of the instruments need to be fully understood, including what constitutes clinically or socially meaningful differences in scores. However, different methods can produce different estimates of minimally important differences.13

    Ranking of providers by score is also problematic because the performance of most providers does not differ much. Such rank orders may have little meaning and can give an artificial distinction between those at the top and those at the bottom.14 It is better to present data in funnel plots (as shown in fig 1) as it avoids ranking and instead focuses attention on any outliers that require local in-depth investigation to determine the reasons for their poor performance.

    A third challenge is the relation between patient experience and outcome. We do not know whether a poor experience increases the likelihood of reporting a poor outcome, or vice versa. This is because the development and use of measures of patients’ experience of humanity has been carried out in isolation from work on patient reported outcomes. Better understanding of the relation is important for interpreting outcomes and experiences.

    Finally, there is the challenge of reconciling patients’ and clinicians’ reports of outcomes. If both report on the same dimension of health (disability) the reports are similar.15 But what if clinicians report an improvement in impairment (such as a better urine flow rate after prostate surgery) but patients report no improvement in their symptoms (still experiencing nocturia four times a night)? This raises the question of the relative importance of the different dimensions of health (impairment, symptoms, disability, quality of life, complications), the answer depending on the perspective of the questioner.

    Developments

    There are several innovative ways in which patients’ reports are, or might be, used. In north west England, patient reported outcome measures for hip and knee replacement are now being linked to providers’ reimbursement, in the hope that such a financial incentive will help improve the quality of clinical care. Providers achieving the best results will get paid a small amount above the standard tariff price.

    So far, use of patients’ views of humanity and outcome have been confined to assessing provider organisations. However, outcome measures may also aid clinical practice. In the US, the Dartmouth primary care cooperative information project (COOP) has developed charts for use as an adjunct to the clinical interview.16 The charts consist of single questions covering areas such as physical fitness, feelings, daily activities, social activities, and overall health. For each question there are five response categories and simple illustrations beside each category to help patients who are marginally literate. Patients and clinicians find such questionnaires helpful.17

    The enormous growth in computing in the past decade has made obtaining data much cheaper.18 Furthermore, the use of computers permits adaptive testing, whereby the questions presented are selected for their particular relevance to that patient.19 Answers to initial questions are used to select other items (questions), reducing the burden for patients of completing items that are not relevant to them. Such systems are likely to improve response rates and speed up data processing. This approach lies at the heart of the US National Institutes for Health patient reported outcomes measurement information system (PROMIS), which is creating “item banks” that measure outcomes seen in various chronic diseases.19 The items will consist of questions from established questionnaires and new ones written by experts and tested in patient populations. It is intended that by using large banks of items, groups of questions can be accurately selected for individual patients, but that scores are comparable across patients.20 The NIH hope this will lead to comprehensive and accurate coverage of patient outcomes for a wide variety of conditions.

    Conclusions

    Traditionally, the person at the centre of health care had no real voice. Professionals judged the quality of services. Now, the patient is central in the hope this will contribute to quality improvement. The impact of the developments discussed above needs to be rigorously evaluated to see if such intentions are realised. The benefits of such changes could revolutionise how health care is evaluated. Patient reporting is not a panacea, but considering patients’ views in a systematic and thoughtful manner is a first step to incorporating them into clinical practice.

    Notes

    Cite this as: BMJ 2009;339:b2495

    Footnotes

    • Contributors and sources: Both authors have considerable experience in developing questionnaires for patients and in collecting and analysing such data. NB is currently working with the Royal College of Surgeons on two audits of elective surgery; CJ has worked on both patient reported outcomes and patient experience of care. Both authors wrote the paper; NB is guarantor.

    • Competing interests: None declared.

    • Provenance and peer review: Commissioned; externally peer reviewed.

    References