Out-of-hours general practitioner (GP) care in Europe is increasingly being handled by large-scale organisations (ie, GP cooperatives), with nurses generally handling the telephone triage. This trend is in response to the increased demand for GP care and attempts to reduce their workload.^1–6 Research shows that the handling of telephone triage by nurses does indeed cause a substantial decrease in the immediate workload and increased efficiency of GPs.^3–7 On the other hand, some studies have shown that subsequent GP consultations increase after nurse-led telephone triage.^7–9

Telephone triage is nevertheless considered the most complex and vulnerable part of the out-of-hours GP care process.^10–12 It has not yet been proven safe, perhaps due to underestimation of complaints.^4,7,10–12 There is no research on the effects of specific education and training on the efficiency and safety of telephone triage. Perhaps that is why it differs in different countries: Denmark opts for GPs² and England and The Netherlands opt for nurses who have a primary- or secondary-care background.^1,3,11

To improve the balance between safety and efficacy the care provided by triage nurses, explicit national telephone guidelines were implemented The Netherlands. The telephone guidelines include a classification system with four levels of urgency.¹⁰

In this study, we examined the ability of triage nurses to adequately estimate the urgency of health problems presented via the telephone, using the telephone guidelines. We also examined the correlations between underestimation of the urgency estimates, educational backgrounds of the triage nurses and their training on the telephone guidelines.

METHODS

Design

A cross-sectional, multicentre, observational study employed five mystery (simulated) patients who telephoned triage nurses in four GP cooperatives. Each of the five mystery patients played four different standardised roles. We chose mystery patients for investigation, because they mimic the reality in care quite reliably and naturally.^13,16

Population

The study was conducted in the last quarter of 2003 and included 118 triage nurses from four GP cooperatives in The Netherlands (box 1). We attempted to obtain a representative picture of the quality of triage in The Netherlands by selecting cooperatives with different periods of existence from different regions with different levels of urbanisation.

Instruments and procedures

A total of 20 Dutch vignettes were written on the basis of practice cases matched to the protocols from the national telephone guidelines. The vignettes represented different complaints, different levels of urgency and different age and sex groups in the most natural manner possible. Life-threatening cases were included slightly more often than in actual daily life. A panel of seven GP experts evaluated the medical accuracy, completeness, representativeness and level of urgency for the vignettes. After modifying the vignettes, the panel of experts again judged the level of urgency for the different vignettes. It was decided a priori that at least six of the seven experts had to assign the same level of urgency to a vignette and that such agreement had to be achieved for >95% of the vignettes. Using this gold standard, only 6 (or 4%) of the 140 judgements (7 experts × 20 vignettes) were found to disagree with regard to the level of urgency for a vignette. Table 1 gives an overview on the vignettes.

The mystery patients were very experienced in their job as simulation patient at the Radboud University Nijmegen Medical Centre and were able to mimic reality quite naturally.^13–16 The five mystery patients were trained to play four vignettes matched with respect to content and emotion. We checked whether the mystery patients played the vignettes naturally and reliably in a pilot study and also by listening to all the audio-taped contacts on a weekly basis.

We made an agreement with the four GP cooperatives with respect to logistics and safety.

The triage nurses were informed of the purpose of the present research and all consented to participate. The triage nurses completed a questionnaire about their education in primary care (2–3 years) or secondary care (4 years). They also provided information on specific training received on the use of national telephone guidelines.

The mystery patients phoned the GP cooperative at random times in out-of-hours. The triage nurses did not know whether the individual on the telephone was a mystery patient and they presumably dealt with the contact in accordance with their own normal routines. At the end of the call or right before the triage nurse went to take action, the standardised patients revealed themselves. They asked the triage nurse her decision regarding the level of urgency on the triage criteria outlined in box 2.

Each of the 20 vignettes was presented 18–20 times equally spread out across the four GP cooperatives, so that each cooperative received the same set.

Consultations that were prematurely terminated and those lacking an urgency estimation were excluded from the analyses.

Analysis

We compared the urgency score assigned by the triage nurse with the gold standard.

We calculated the sensitivity, specificity and predictive values of the estimated urgency. For this purpose, a dichotomy was created between the U1–U2 life-threatening or potentially life-threatening categories of urgency and the remaining U3–U4 categories.

Next, we analysed the underestimation of urgency in the subset U1–U2 life-threatening cases. We carried out a multilevel logistic regression analysis, with underestimation of urgency as the dependent variable and educational background and hours of training on the national telephone guidelines as independent variables, with the GP cooperatives as a random factor. A significance level of p<0.05 was adopted.

RESULTS

The mystery patients made a total of 370 telephone contacts, with 352 of the contacts proving usable in the end. A total of 18 contacts were dropped mostly due to no urgency estimate (n = 5) or premature termination due to recognition by the mystery patient (n = 13).

For 242 (69%) of the 352 telephone contacts, the urgency estimation was in perfect concordance with the gold standard. For the 110 non-concordance contacts, the urgency estimate of 106 (30%) contacts differed not more than 1 point, and for 4 (1%) of the contacts, the urgency estimates differed ⩾2 points from the gold standard.

For 44 (12.5%) of the telephone contacts the triage nurses scored higher urgency, and for 66 (19%) of the contacts the triage nurses scored lower urgency than the gold standard (table 2).

The capacity of the triage nurses to discriminate (potentially) life-threatening cases from less urgent cases was examined through comparison of the U1 and U2 figures with the U3 and U4 figures. The sensitivity was 0.76 (63/83) and the specificity was 0.95 (256/269). The positive predictive value of the estimates of triage nurses was 0.83 (63/76). This is therefore much higher than the 0.24 ((35+48)/352) a priori probability of a life-threatening problem. The negative predictive value of the estimates of triage nurses was 0.93 (256/267), while the a priori probability of a non-life-threatening problem was 0.76 (269/352).

The triage nurses who were trained on the use of national telephone guidelines had a lower rate of underestimation of the urgency (OR = 0.10, 95% CI 0.01 to 0.81). The educational background (primary or secondary care) of the nurses had no significant relationship with the rate of underestimation.

DISCUSSION

The urgency judgements made by the triage nurses after telephone contact with a mystery patient were found to be in two-thirds agreement with the gold standard. The sensitivity and positive predictive values were not high, which meant that triage nurses tended to underestimate urgent complaints. Conversely, the specificity and negative predictive values were high, which meant that the triage nurses delivered efficient care.

We found a significant correlation between the accurate estimation of the urgency and specific training on telephone guidelines. This finding suggests that training can help improve the safety of care, but we do not know what is the required intensity or frequency of this training.

The use of mystery patients in practice mimics reality in an outstanding way. However, despite our efforts, one cannot prevent subtle differences in the presentations of the different complaints by the standardised mystery patients.

Additional research is needed to unravel the different determinants of the quality and safety of telephone consultations with triage nurses. A follow-up study in general practice reviewing safety patient contacts with the GP cooperative is also recommended.¹⁸

In The Netherlands, there is a trend to employ a special telephone doctor to supervise about 4–6 triage nurses.^10,11 Investigation of this role is needed: does it improve the quality and safety of telephone triage?

In the delivery of high-quality care, safety should always take priority over efficiency because of the potential severe consequences for the patient.^10–12 This research proves that telephone triage by nurses is efficient, but is possibly not safe.

Should we be worried about the outcome of this study? Should we remove the nurses from their triage tasks and should doctors perform telephone triage as is done in Denmark?² Other studies have shown that telephone triage by nurses is safe.^4–7,19 Moreover, there is no research comparing doctors and nurses performing triage. In addition, we cannot conclude that nurse triage is less safe than GP triage, because that requires another study design. Perhaps it does not matter who performs the triage because the telephone as a medium is unsafe.

Indeed we should worry about the safety of telephone triage and take major steps to improve this. GP cooperatives should adopt safety rules such as: “When patients ring for the second time you should arrange a face-to-face meeting with the doctor.” They should also encourage the attitude of not being too restrictive in arranging face-to-face contact, because the telephone is perhaps an unsafe medium.^11,12,17

We recommend an educational certified programme for triage nurses and a direct second safety check of all cases by a specially trained GP, who supervises telephone triage nurses.^10,11 Further, the use of computerised decision support may also be helpful to enhance the safety of telephone triage.^4,7,20–22 Finally, we recommend analysis of medical (near)calamities in peer group meetings.