Primary Care

An observational study comparing quality of care in walk-in centres with general practice and NHS Direct using standardised patients

BMJ 2002; 324 doi: http://dx.doi.org/10.1136/bmj.324.7353.1556 (Published 29 June 2002) Cite this as: BMJ 2002;324:1556
  1. Clare Grant, clinical lecturera,
  2. Ruth Nicholas, research associatea,
  3. Laurence Moore, senior research fellowb,
  4. Chris Salisbury (c.salisbury{at}bristol.ac.uk), senior lecturera
  1. a Division of Primary Health Care, University of Bristol, Bristol BS6 6JL
  2. b Cardiff School of Social Sciences, University of Cardiff, Cardiff CF10 3WT
  1. Correspondence to: C Salisbury
  • Accepted 4 April 2002

Abstract

Objectives: To compare the quality of clinical care in walk-in centres with that provided in general practice and by NHS Direct.

Design: Observational study involving assessment of clinicians by standardised patients.

Setting: 20 walk-in centres, 20 general practices, and 11 NHS Direct sites.

Participants: 297 consultations with standardised patients, 99 in each setting, carried out by professional role players trained to play five clinical scenarios (postcoital contraception, chest pain, sinusitis, headache, and asthma).

Main outcome measures: Primary outcomes were mean scores on consensus derived checklists of essential items for the management of the clinical scenarios. Data were also collected on access to and referral by walk-in centres, general practices, and NHS Direct.

Results: Walk-in centres achieved a significantly greater mean score for all scenarios combined than general practices (difference between groups 8.2, 95% confidence interval 1.7 to 14.6) and NHS Direct (10.8, 5.5 to 16.1). There was considerable between scenario variation, with walk-in centres performing particularly well on postcoital contraception and asthma scenarios. In contrast to general practices, walk-in centres and NHS Direct referred a higher proportion of patients (26% and 82%, respectively).

Conclusion: Walk-in centres perform adequately and safely compared with general practices and NHS Direct for the range of conditions under study, but the impact of referrals on workload of other healthcare providers requires further research.

What is already known on this topic

What is already known on this topic Assessment of the quality of care provided by walk-in centres is an important part of their overall evaluation

The one previous study to assess quality in this setting reported negative findings but was methodologically flawed

Standardised patients are a valid and reliable method of assessment of quality of clinical care

What this study adds

What this study adds Walk-in centres perform adequately and safely compared to general practice and NHS Direct for the range of conditions under study

Introduction

The UK government, as part of a bid to modernise health services and to make them more convenient to use, has introduced NHS walk-in centres and commissioned their evaluation by an independent team. Our study forms one component of that evaluation. We aimed to determine whether walk-in centres, staffed mainly by nurses, provide adequate and safe clinical care to a range of patients, and how quality of care in these centres compares with that provided in general practice and by NHS Direct.

The only previous evaluation of UK walk-in centres found quality of care disappointing overall, some care positively dangerous, and considerable variation between sites.1 The study considered just eight centres, with no comparative group and several other flaws in its methods.2

Several methods exist for evaluating quality of clinical care. Systematic biases are associated with indirect methods, such as review of medical records or self-reporting by clinicians.3 Direct methods of observation, such as video recording, require large numbers of consultations to be reliable, because of the variable case mix seen in clinical practice. In addition, clinicians may change their behaviour when observed. The use of covert standardised patients for quality assessment is gaining favour because it avoids these methodological problems.

Standardised patients are people trained to portray a clinical scenario for teaching or research purposes. They have been used in the United States for over 30 years and are the subject of a major body of educational research. 4 5 Several studies have employed them to explore what occurs in clinical practice, including studies concerned with quality assurance.611 Whereas this use of standardised patients is novel in the United Kingdom, experience from elsewhere suggests it is a valid and reliable approach.12 We used standardised patients to compare care in walk-in centres with that in the primary care settings of general practice and NHS Direct: we focus on what is necessary, rather than ideal, with respect to patient care.

Participants and methods

Recruitment of clinical sites

For logistical reasons we approached sites in three geographical areas, in and around Bristol, Birmingham, and London. We invited general practices involved in research or teaching to participate, and we approached walk-in centres and NHS Direct sites on the basis of their geography. Twenty out of 25 (80%) walk-in centres, 11 out of 12 (92%) NHS Direct sites, and 24 out of 62 (39%) practices agreed to participate: we included the first 20 practices to respond. Twelve practices (19%) declined to take part, and the remainder did not respond.

Ethical approval for our study was obtained from a multicentre research ethics committee. Participating sites all gave informed consent for contact with standardised patients but were unaware when they would consult. Participating sites were ensured anonymity: outcome data were presented such that individuals could not be identified.

Selection of clinical scenarios

We chose five clinical scenarios, largely to represent problems likely to be presented by patients to walk-in centres (box 1). The scenario on postcoital contraception was intended to assess management of a common, straightforward problem, the scenario on chest pain to assess ability to exclude a potentially serious diagnosis and reassure accordingly, and the scenario on sinusitis to assess issues around antibiotic prescribing. The scenario on headache was devised to assess ability to explore psychosocial issues, and the scenario on asthma to assess the history taking on drugs and awareness of the side effects of the drugs.

Box 1: Clinical scenarios portrayed by standardised patients

  1. A 23 year old woman requesting postcoital contraception

  2. A 30 year old man with musculoskeletal chest pain

  3. A 35 year old woman with symptoms of sinusitis suggesting a bacterial cause

  4. A 27 year old man with tension headache and underlying depression

  5. A 30 year old man with worsening asthma caused by over the counter ibuprofen

RETURN TO TEXT

Derivation of assessment criteria

We assessed clinical care against prospectively determined standards. We constructed lists of essential criteria for the adequate management of each scenario by a stepwise procedure, based on the Delphi process.13 The research team identified potential criteria for essential actions relevant to the five scenarios, by review of relevant literature and consensus based guidelines. We posted lists of potential criteria to a consensus panel of nine general practitioners and five nurse practitioners. Criteria were included as items in the final checklists if, after two rounds, 93% or more members of the panel agreed they were essential. Checklists consisted of eight to 17 items, depending on scenario and setting, with items grouped into “history taking,” “examination,” and “diagnosis, advice, and treatment.” For each scenario, checklists for the walk-in centres and general practices were identical, with the NHS Direct checklist containing all but the examination items.

Standardised patient consultations

Fifteen professional role players involved in medical education played standardised patients. They were chosen to match the roles assigned to them in terms of age and sex. Five role players, each portraying one scenario, worked in each locality. Each role player visited a particular walk-in centre or general practice once, but owing to the smaller number of sites for NHS Direct, contacted one NHS Direct site up to three times. Overall, 305 contacts were planned, 100 in walk-in centres, 100 in practices, and 105 with NHS Direct. Consultations took place over a 13 week period from July to September 2001. Standardised patients visited the general practices as temporary residents to avoid the logistical problems of registration as a new patient. They completed checklists immediately after consultations.

Training of role players

The role players were provided with detailed information about the sites and personal data relating to their roles. They received a day's training in role portrayal and clinician assessment, during which data on accuracy of portrayal and reliability of assessment were collected.

Accuracy of portrayal of standardised patients

Accuracy of portrayal, defined as the proportion of predefined clinical features presented correctly, was measured by two observers independently completing a performance item checklist during the final round of role playing.14 Across all scenarios, 1038 of 1164 (89%) clinical features were portrayed correctly. Consistency of portrayal was maintained by role players watching a videotape of their performance before each consultation during the study.

Reliability of assessment by standardised patients

Reliability of assessment was measured by comparing role players' responses to each checklist item with gold standard responses. These were the responses of two general practitioners who independently watched videotaped consultations from the training day and completed the checklists blind to role players' responses. Across all scenarios and role players κ=0.8. The reliability between role players was measured by the role players independently assessing each other's videotaped consultations. The responses of two role players observing the same videotaped scenario were compared. Across all scenarios and role players κ=0.9. The reliability of the role player over time was measured by role players assessing their training day videotapes half way through the study and comparing responses to those on the training day: κ=0.7.

Validity of standardised patients

Participating clinicians were encouraged to contact the study team, anonymously via an answer machine, if they suspected a consultation with a standardised patient. Twenty three calls were made—five about true standardised patients and 18 about true patients. Thus the detection rate was 1.7% (5 of 297), and the positive predictive value was 21.7% (5 of 23).

Analysis

For each consultation we calculated a score representing the proportion of essential criteria fulfilled for all items and separately for the three subgroups of items (history taking, examination, and diagnosis, advice, and treatment). We calculated the means of the all item and subgroup scores for each of the three primary care settings, with 95% confidence intervals calculated with design weighted survey estimators. We then undertook estimation of differences between mean scores for the three settings by using multivariable regression models, with setting and scenario included as fixed effects. We accounted for non-independence of repeat observations within individual centres in two alternative ways: models were estimated with design weighted survey estimators and with individual centre included as a random effect, as implemented in Stata.15 For each model we then added the interaction between primary care setting type and scenario and tested for significance. Where this interaction was significant we estimated separate models excluding first NHS Direct consultations,(allowing comparison of walk-in centres and practices) and second general practice consultations (allowing comparison of walk-in centres and NHS Direct). Within each scenario we estimated mean scores, mean score differences between settings, and 95% confidence intervals for these statistics by using design weighted survey estimators. Scores for the three subgroups of items were not normally distributed, so we repeated all analyses with the Wilcoxon rank sum test. In no case was the result of the non-parametric test significantly different to the analyses presented in tables 1 and 2. Finally, we calculated the variation in scores (standard deviation) within each setting.

Table 1.

Mean scores (95% confidence intervals) on all essential items, with comparisons between walk-in centres and general practice and walk-in centres and NHS Direct

View this table:
Table 2.

Mean scores (95% confidence intervals) on history taking, examination, and diagnosis, advice, and treatment items with comparisons between walk-in centres and general practices and walk-in centres and NHS Direct

View this table:

The results of the design weighted models and random effects models were similar in all cases, and thus only the design weighted analyses are presented as these tended to be slightly more conservative than the random effects models. For the models based on all items and those restricted to history taking and diagnosis, advice, and treatment, the interactions between primary care setting and scenario were significant. Thus, we present the results of these analyses and analysis of examination items separately for comparisons between walk-in centres and general practice and then between walk-in centres and NHS Direct.

Results

Data were collected on 297 of the planned 305 consultations, 99 in each setting. One consultation in general practice and six with NHS Direct were not completed owing to problems with access, and data on one consultation were lost in the post. Most (91%) consultations in walk-in centres were exclusively with a nurse, whereas most (96%) in general practice were with a doctor. A minority (3%) of consultations with NHS Direct involved a call handler rather than a nurse.

Quality of care

Walk-in centres versus general practice

Considering all scenarios together, walk-in centres achieved a significantly greater mean score for all essential items than did general practices, but there were between scenario differences. Quality of care for postcoital contraception and asthma was significantly better in walk-in centres than in general practices, that for sinusitis and headache was similar in the two settings, and that for chest pain was better in general practice, although not significantly so (table 1).

Overall, and for each scenario, walk-in centres scored better on history taking, with significant differences overall and for two individual scenarios (table 2). Overall, and for the two scenarios involving examination (chest pain and asthma), general practices scored better on examination, although for asthma the difference was not significant. Overall, there was no significant difference between quality of diagnosis, advice, and treatment provided by walk-in centres and general practices. However, for postcoital contraception and asthma, the quality of diagnosis, advice, and treatment was significantly better in walk-in centres, and for sinusitis it was significantly better in general practices. In summary, walk-in centres provided equivalent if not better quality of care than general practice, with the exception of advice and treatment of sinusitis and examination of chest pain.

Walk-in centres versus NHS Direct

Considering all scenarios together, walk-in centres achieved a significantly greater mean score for all items than did NHS Direct, again with between scenario differences. Quality of care for postcoital contraception and asthma was significantly better in walk-in centres, with no significant difference for other scenarios (table 1). Overall, and for postcoital contraception and asthma, walk-in centres scored significantly better on history taking, with the same pattern for diagnosis, advice, and treatment (table 2).

Referral

About a quarter (26%) of consultations in walk-in centres and four fifths (82%) with NHS Direct resulted in referral (table 3). Patients were advised to go to an emergency department in 5% of consultations in walk-in centres and 13% with NHS Direct, but not from consultations in general practice.

Table 3.

Percentages (numbers) of referrals overall and to emergency departments by scenario and setting

View this table:

Between site variability

Considering all scenarios, the variability of quality scores, expressed as standard deviation, was highest for NHS Direct and lowest for general practice (table 4). However, variability of history taking for all scenarios was higher in general practice. Variability of quality in the three settings varied across scenarios. NHS Direct's high variability was largely the result of the postcoital contraception scenario, where referral may have substituted for consistent performance on essential items.

Table 4.

Standard deviations for mean quality scores by scenario and setting

View this table:

Access

Contacting NHS Direct was sometimes time consuming and unsatisfactory. About one quarter (25 of 99) of completed calls involved “call back” (mean wait 33 minutes). Three consultations with NHS Direct were not completed owing to length of wait for “call back.” Fewer problems were reported accessing general practice and walk-in centres.

Discussion

Walk-in centres provided equivalent if not better quality clinical care than both general practice and NHS Direct, for the range of clinical conditions assessed. Substantial variation existed in quality between scenarios, with walk-in centres performing particularly well for postcoital contraception and asthma. They performed significantly less well than general practice in examination of chest pain and diagnosis, advice, and treatment of sinusitis. These inadequacies do not necessarily provide evidence of poor quality care. Firstly, examination may be of secondary importance in the overall management of musculoskeletal chest pain. Secondly, significantly lower antibiotic use by walk-in centres for sinusitis may reflect acknowledged difficulties identifying patients who will benefit from antibiotics. Walk-in centres' better performance was particularly noticeable for history taking, perhaps owing to the longer consultations undertaken in this setting.16

Considerable variability exists in the quality of care provided by individual sites. Variability among walk-in centres is greater than among practices, possibly due to variation in training, experience, and competencies of nurses working in walk-in centres.17 However, variability of history taking is greater in general practice, perhaps owing to the use of clinical assessment software in walk-in centres.

Although walk-in centres provide better quality of care, both the centres and NHS Direct referred a proportion of standardised patients elsewhere. Some of the referrals were to emergency departments, which may not be appropriate for the conditions studied and raises questions about the impact on workload of other providers, as addressed elsewhere in the evaluation.

Comparison with other studies

The accuracy of portrayal (90%) by standardised patients compared well with previous studies, as did their reliability (κ=0.7 to 0.9). 14 18 Detection (1.7%) and positive predictive value (21.7%) also compared favourably.1821

Although checklists were meant to consist of essential items, the proportions of items achieved in all three settings were low, a finding consistent with another study that assessed general practitioners against peer determined standards.22 Performance may not reflect competence, as clinicians exhibit efficiency by only carrying out what is necessary at a particular moment.22

Methodological issues

The study's main limitations are non-random sampling of participating sites, use of a limited number of scenarios—some more discriminating than others—and use of novel assessment checklists. Participating sites, particularly general practices, were likely to be more interested in the research question and may have provided a higher quality of care, possibly attenuating the study findings. Scenarios were chosen as typical of those seen in walk-in centres and because they were appropriate for portrayal by standardised patients. Scenarios necessitating the presence of abnormal findings or potentially involving certain types of physical examination or referral to third parties could not be included. The finding that walk-in centres offer safe care cannot necessarily be extrapolated to all clinical conditions, although we aimed to assess important, generalisable aspects of clinical care.

Box 2: Service models for the three primary care settings

Walk-in centres

These offer assessment, advice, and treatment for minor illness and injuries. Most consultations are with nurses who use care protocols and clinical assessment software. The average consultation length is 14 minutes.16

General practice

Most consultations are with doctors who do not routinely use care protocols. The average consultation length is 9 minutes.24

NHS Direct

This is a telephone advice service. Consultations are with nurses, who use clinical assessment software similar to that used in NHS walk-in centres. The average consultation length is 14 minutes.25

The ability of the scenarios to discriminate between settings was variable. Scores on the scenario for postcoital contraception were highly variable both overall and for NHS Direct in particular, perhaps because NHS Direct's management of this scenario always involved referral. Consequently, performance on history taking and advice and treatment may have been less thorough.

Scenarios did not involve children and elderly patients, who attend general practice most often. Nor did they assess some of the supposed strengths of general practice—for example, continuity and availability of past medical records. Also, the methodology, requiring all standardised patients to be temporary residents, did not lend itself to the assessment of some of these strengths. Thus, the findings do not suggest care in general practice is inferior to walk-in centres, rather that walk-in centres perform adequately and safely compared with controls.

The development of assessment checklists used by standardised patients in previous studies has received little attention, although their construction is crucial to the reliability and validity of assessment.23 Checklists used in this study seem to have high face validity and content validity, although few data were collected on other properties, such as reproducibility.

Conclusion

Standardised patients have a useful role in the assessment of quality in the UK primary care setting. Also, walk-in centres provide adequate, safe clinical care to a range of patients, compared with general practice and NHS Direct. However, the cost effectiveness of walk-in centres and their impact on workload of other healthcare providers requires further assessment.

Acknowledgments

We thank the general practices, walk-in centres, and NHS Direct sites that took part in the study, the role players for their participation, members of the consensus panel for their input, and Judi Laister, Steve Harvey, Norma Jones, and John Pollock for their assistance.

Contributors: CG developed the design and content of, and supervised, the study and wrote the first draft of the manuscript. RN assisted with development of the study, and managed it on a day to day basis. LM performed the statistical analyses and revised critically the final manuscript. CS had the idea and wrote the original protocol for the study and revised critically the final manuscript. CG will act as guarantor for the paper. The study was carried out on behalf of the National Evaluation of Walk-in Centres Team.

Footnotes

  • Funding This research was conducted independently by the University of Bristol, funded by the Department of Health. The views expressed in this publication are those of the authors and not necessarily those of the Department of Health.

  • Competing interests None declared.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
View Abstract