Intended for healthcare professionals

General Practice

Reliability and validity of a new measure of patient satisfaction with out of hours primary medical care in the united kingdom: development of a patient questionnaire

BMJ 1997; 314 doi: (Published 18 January 1997) Cite this as: BMJ 1997;314:193
  1. Robert K McKinley, senior lecturera,
  2. Terjinder Manku-Scott, research associatea,
  3. Adrian M Hastings, lecturera,
  4. David P French, research associatea,
  5. Richard Baker, director, Eli Lilly national clinical audit centrea
  1. aDepartment of General Practice and Primary Health Care University of Leicester Leicester General Hospital Leicester LE5 4PW
  1. Correspondence to: Dr McKinley
  • Accepted 22 November 1997


Objective: To develop a reliable, valid measure of patient satisfaction with out of hours care suitable for large scale service evaluation.

Design: Focus group meetings and semistructured interviews with patients to identify issues of importance to patients and possible questionnaire items; interviews and two pilot studies to test and identify new questionnaire items; modification or removal of items to eliminate ambiguity and reduce non-response and skewed responses; questionnaire survey of out of hours care.

Setting: Greater Manchester and Leicester.

Subjects: 11 general practice patients participated in the focus groups and 28 in the semistructured interviews; 41 in the preliminary interviews; 41 and 378 in the postal pilots; and 1466 in the survey of out of hours care.

Results: A 32 item questionnaire was developed. Component analysis indicated seven scales (satisfaction with communication and management, doctor's attitude, continuity of care, delay until visit, access to out of hours care, initial contact person, telephone advice) related to overall satisfaction and containing issues identified as important to patients. Levels of reliability were satisfactory, Cronbach's α correlation coefficient exceeding 0.60 for all scales.

Conclusion: A reliable, valid measure of patient satisfaction has been developed, suitable for large scale evaluation of out of hours care.

Key messages

  • The provision of out of hours primary medical care is changing, and these changes need to be evaluated and monitored

  • Patient satisfaction is an important measure of the outcome of health care

  • A reliable and valid measure of patient satisfaction with out of hours primary medical care has been developed

  • Development of such scales is demanding on time and experience but is feasible

  • Ad hoc measures of satisfaction should be avoided and when possible reliable, valid scales used


During the past 30 years many general practitioners have stopped providing personal 24 hour care to patients and have subcontracted much of it to commercial deputising services.1 This withdrawal from the personal provision of out of hours care has been fuelled by rising demand2 which may be inappropriate,3 4 5 fatigue,6 stress,7 and concerns about personal safety.8 Contractual arrangements have been changed to allow general practitioners greater freedom to choose how they provide out of hours care.9 Supported by additional public funding, this has encouraged the development of various models of care, including cooperatives and out of hours centres. These should be evaluated to ensure that both the quality of care and practitioner wellbeing are maintained or improved. Quality is multifaceted10 and its assessment requires multiple measures of process, such as response times, telephone advice rates, prescribing, and admission rates combined with measures of outcome such as health status and satisfaction.11 12 Low patient satisfaction may result in poor compliance with the potential for waste of resources and suboptimal clinical outcome.13 14 Satisfaction of the legitimate demands of patients is therefore an objective of all medical care15 and should be included as an outcome measure.16

Measuring patient satisfaction with medical care is not straightforward. One approach is to use qualitative methods,17 18 but these are difficult to use for routine large scale service evaluation. An alternative is to use a quantitative questionnaire. Such a questionnaire must be reliable19–that is, the random error of responses must be minimised so that consistency of measurement is achieved. The questionnaire must also be valid–that is, it must be a true measure of what it purports to measure and must not be subject to bias.20 Validity can further be characterised as face, content, criterion, or construct validity.19 Much of the early work on measuring patient satisfaction took place in the United States21 but patient satisfaction questionnaires for care provided by United Kingdom general practitioners have been published. These assess satisfaction with the practice,22 access to doctors,23 individual consultations,24 and advice given in the consultation.13 Though other workers have reported patient satisfaction with out of hours care, they did not use questionnaires with established reliability and validity.25 26 27 28

We report the development of a reliable and valid questionnaire which can be administered by interview or completed by the patient or carer for measuring patient satisfaction with out of hours primary medical care in the United Kingdom.

Subjects and methods

Identifying issues important to patients

We used qualitative methods to identify issues about out of hours care important to patients and so develop questionnaire items. We invited patients to join two focus groups29 led by a non-clinical colleague. They were recruited from general practice registers and community groups to represent a range of patients from parents and guardians of children to elderly people from different ethnic, cultural, and social backgrounds. We included parents and guardians of children because they initiate many requests for out of hours care.5 30 Group meetings were audiotaped and coded separately by two of us (RKM and AMH). We used this material to inform the content of semistructured interviews administered to patients or their carers who had recently requested out of hours care from two large city practices or their deputising service. In the interviews we further explored the issues important to patients and tested potential questions for inclusion.

We compiled a list of elements of patient satisfaction relevant to out of hours care from unpublished questionnaires (D Wilkin, personal communication, 1993) and questionnaires used in studies of out of hours care in the United Kingdom.23 25 26 27 28 We compared this list with the issues identified during our qualitative work to check that we had considered all those previously identified and thus ensure content validity. To further ensure content validity we asked all patients interviewed during development of the questionnaire to comment on its content and suggest additional issues or questions.

Questionnaire development

We developed a bank of questions to enable us to produce multi-item scales, which are more reliable than single questions.21 We selected 47 positively and negatively worded questions which covered the topics important to patients for use in the preliminary questionnaire. We used a balanced Likert five point scale (strongly agree, agree, neutral, disagree, strongly disagree) to record responses. We administered the questionnaire by interview to 41 patients who had recently requested out of hours care from the same practices. Questions which were confusing, ambiguous, or gave very skewed responses were either removed, rewritten, or replaced.

Questionnaire refinement

We undertook two further postal pilots, the first with a 48 item questionnaire administered to patients from six practices in the city and suburbs of Leicester and the second with a 34 item questionnaire administered to patients from the six Leicester practices and one practice in Manchester. Consecutive patients or carers who had requested out of hours care were sent a questionnaire within 72 hours of the request and a self addressed envelope for return. We used spss for all statistical analyses. After each pilot, questions with highly skewed responses or high non-response rates were removed or rewritten.

Principal components analysis31 with varimax rotation was used to indicate which questions examined similar aspects or components of out of hours care. The eigenvalue limit for the principal components analysis was set at one. We retained the issues patients had identified as important, thus maximising our chances of achieving content validity. We omitted from the principal components analysis questions relating to overall satisfaction, as we anticipated that all questions would tend to load with this underlying general component.21 24 We also omitted responses from patients who received telephone advice only and therefore did not answer the questions related to receiving a visit.

We calculated component scores by scoring questions from one to five (five always representing maximum satisfaction), summing them, and expressing the total as a percentage of the maximum possible score for the component. If a respondent omitted half or more of the questions in a component we excluded these data from analysis. By calculating Cronbach's α coefficient we estimated the internal consistency or reliability of each component. When necessary we added questions to improve the reliability of components. Evidence of construct validity was sought by calculating a matrix of Pearson's correlation coefficients containing components and the overall satisfaction scale.

Evaluation of questionnaire

The final version of the questionnaire contained 32 questions. It was used in a comparative trial of out of hours care provided by deputising services and practice doctors in 11 practices in Greater Manchester and three in Leicester.11 12 The questionnaire was self completed by the patient or carer during an interview 24 to 120 hours after a request for out of hours care. Two hundred consecutive patients recruited to the study or their carers were asked to complete a second questionnaire later the same day and return it by post as a test of test-retest reliability.

We analysed these data using the methods described under questionnaire refinement. To recheck content validity we asked 12 general practitioners, three practice nurses, and eight colleagues who were not otherwise involved in the development of the questionnaire in the departments of general practice in Manchester and Leicester to review the components indicated by the principal components analysis. They judged whether these were coherent and reflected the issues important to patients and also suggested names for each component and the two groups of questions related to overall satisfaction and telephone advice. We rechecked construct validity by calculating the intercomponent correlations. We rechecked reliability with Cronbach's α coefficient and checked test-retest reliability by calculating the regressions of the retest on the test data and Pearson's correlation coefficient between the scores for each scale.


Eleven people participated in the focus groups. Seven were female, six were from ethnic minorities, five were parents or guardians of children, three were adults with chronic illnesses, and three were aged over 65. One came from a rural area and five from inner city areas. Table 1) gives the age, sex, and ethnic origin of patients who participated in the development of the questionnaire. Of the 1466 patients in the comparative trial, a questionnaire was completed for 1402 (95.6%) and, of these, 163 received telephone advice only. The median (interquartile range) completion rate for questions was 96.5% (95.7% to 97.1%). Scale scores were calculated for a median (interquartile range) of 97.7% (94.5% to 98.1%) of responses. A total of 112 (56.0%) of 200 retest questionnaires were returned.

Table 1

Age, sex, and proportions of patients who described their ethnic origin as white. Figures are numbers (percentages) of patients who participated in development of questionnaire

View this table:

The box lists the issues identified by the focus groups and patient interviews together with the elements of patient satisfaction with out of hours care identified from previous studies.23 25

Components of patient satisfaction with out of hours care in United Kingdom identified by focus groups and interviews with patients and by literature search

Focus groups and interviews

(1) Access to out of hours care

Ease of requesting care, telephonist's attitude/understanding, ease of getting a doctor to visit, ease of getting telephone advice, information about delays/what would happen next, delay until doctor telephoned/visited

(2) Interpersonal aspects

Doctor concerned/dismissive, courteous/rude, patient/inconsiderate, sympathetic/abrupt, friendly, caused patient to feel embarrassed or guilty, no choice of doctor, continuity of care

(3) Quality of care

Communication/history/examination, experienced/competent doctor, explanation/prescription, advice about follow up, time spent in consultation

(4) Outcome Felt better/worse afterwards, doctor's response not useful

(5) Overall satisfaction

Data from other United Kingdom studies of out of hours care23 25 26 27 28

Access to out of hours care

Ease of contact, telephonist's attitude, prior knowledge of visiting doctor, availability of telephone advice, delay to visit

Interpersonal aspects

Doctor's manner, gave impression call was unnecessary, doctor understood problem, doctor's command of English

Quality of care

Communication with doctor, physical examination, explanation of problem, treatment/medication, prognostic information


Overall satisfaction

The principal components analysis of the 1402 completed questionnaires from the trial of out of hours care identified six components. Each was judged to be coherent and to represent a separate scale related to satisfaction with out of hours care by the independent reviewers, who also identified titles for each scale. The scales were satisfaction with communication and management (seven questions), doctor's attitude (five questions), continuity of care (four questions), delay until visit (three questions), access to out of hours care (three questions), the initial contact person (two questions), telephone advice (four questions), and overall satisfaction (four questions). The appendix lists the questions in each scale and their titles together with their Cronbach α coefficients, the means and standard deviations of the scale scores, and the variance explained by each scale. The loading of each question on the component to which it was assigned is shown by the coefficients from the rotated factor matrix. The regressions for the retest on the test data and their correlations are shown in table 2). The regressions were all less than one, though (with the exception of “Initial contact person”) Pearson's correlation coefficients were in the range 0.72 to 0.86.

Table 2

Gradient and 95% confidence intervals and constant terms for regressions of “retest” on “test” scores together with Pearson's correlation coefficients and 95% confidence intervals for their correlations

View this table:

Additional evidence which confirmed the construct validity of the questionnaire was provided by the correlation matrix for the eight scales (table 3). All tended to be more highly correlated with overall satisfaction than the others. Scale 3 (continuity of care) had the lowest interscale correlations. There was no difference in the factor structure disclosed by the principal components analysis when the results of patients in Manchester and Leicester were analysed separately.

Table 3

Matrix of correlation coefficients between all scales (upper left portion of table) and number of scores included in each interscale comparison (lower right portion of table)

View this table:


These findings indicate that this questionnaire has satisfactory reliability and validity. It can detect different levels of satisfaction12 and is therefore suitable for evaluating out of hours care received by a broad range of patients. The questionnaire has satisfactory internal reliability with Cronbach's α coefficients greater than 0.60 for all scales and greater than 0.70 for five.38 The test and retest scores were highly correlated, though the regressions show that the retest scores were generally lower, so that there may have been a real fall in satisfaction with time. In a true test of test-retest reliability the variable and measurement technique should be the same on both occasions. The lower retest scores may therefore also reflect the difference in the method of application, with greater expressed satisfaction when the research assistants were present. Nevertheless, these data indicate that the retest reliability of the questionnaire is broadly satisfactory.

Content validity was ensured by the process of questionnaire development. Issues important to patients identified during the qualitative phase included all elements identified from other studies and also additional issues. Content validity was initially shown by the outcome of the development interviews and the failure of patients to identify additional issues. Further evidence of content validity came from the outcome of the principal components analysis and was confirmed by people independent of the development team. The interscale correlations show that, though each scale is correlated with and hence related to overall satisfaction, the scales assess different aspects of satisfaction and contribute to the global measurement of satisfaction, a finding which argues in favour of construct validity. Future evaluations of this questionnaire should further examine construct and criterion validity.

The interscale correlations were lower for continuity of care (scale 3) than for the other scales. This is in contrast with the importance of continuity of care to overall satisfaction with a practice.39 40 Choice of doctor and continuity were identified as issues important to patients in the focus groups and interviews. Nevertheless, when patients or carers believe they need to see a doctor immediately they may place greater value on the availability of care than whether or not they see a familiar doctor. We developed the questionnaire to evaluate satisfaction with domiciliary out of hours care and did not include questions about the environment in which care was provided. This will have to be included in evaluations of out of hours centres if it is important to patients.

The acceptability of the questionnaire to patients is shown by the high response rates for each question (median 96.5%) and the high proportion of responses for which we could calculate scale scores. We achieved response rates of over 50% in the postal pilots in most practices with a single mailing. This shows that the instrument can successfully be administered at interview and probably by post to a broad range of urban patients. The range of scores obtained shows that a well developed questionnaire does not necessarily indicate high levels of satisfaction and suggests that the questionnaire can detect differences in satisfaction.

We have developed a questionnaire of proved reliability and validity which is acceptable to patients. Further development of the questionnaire in other settings with other patient populations is desirable. Development of reliable, valid questionnaires demands time and expertise but is feasible. With the increasing development of such instruments for general practice in the United Kingdom13 22 23 24 it is no longer acceptable to use ad hoc measures.

General practitioners are disillusioned with out of hours care.41 42 Nevertheless, patients have a right to timely, appropriate, and humane medical care. New models of care are being developed and both new and existing models should be evaluated or audited to ensure that the needs of both patients and practitioners are met. This will require measurement of patient satisfaction. Overall evaluations will require a judgment about the relative importance of each need, but none should be ignored by any who use, provide, or pay for the service being evaluated. We therefore encourage all who wish to evaluate an out of hours service to include assessment of patient satisfaction.


We thank the patients who participated in the focus group meetings and the initial face to face interviews; colleagues who provided copies of unpublished questionnaires; the practices and patients who participated in the studies; Hilary Hearnshaw, who led the focus groups; Alison Cooper, for advice on the design of the questionnaire; and the practice nurses, general practitioners, and colleagues in Leicester and Manchester who reviewed the questionnaire.

Funding: MRC Health Services Research Board. Service support for the participating practices was provided by Trent and North Western Regional Health Authorities.

Conflict of interest: None.


Scales devised by principal components analyses of out of hours satisfaction questionnaire, their α coefficients, mean and standard deviation of scale scores, and percentage of variance in principal components analysis explained by each scale. Questions about general satisfaction and satisfaction with telephone advice were omitted from principal components analysis. Coefficients from rotated factor matrix with Kaiser normalisation are shown. Question numbers represent order in questionnaire. Plus and minus signs indicate whether question is positively or negatively worded

View this table:


  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.
  30. 30.
  31. 31.
  32. 32.
  33. 33.
  34. 34.
  35. 35.
  36. 36.
  37. 37.
  38. 38.
  39. 39.
  40. 40.
  41. 41.
  42. 42.