Published 29 September 2009, doi:10.1136/bmj.b3851
Cite this as: BMJ 2009;339:b3851

Research

Reliability of patient responses in pay for performance schemes: analysis of national General Practitioner Patient Survey data in England

¹ Department of Public Health and Primary Care, University of Cambridge, Institute of Public Health, Cambridge CB2 0SR, ² RAND Corporation, 1776 Main Street, Santa Monica, CA 90401-3208, USA, ³ Ipsos MORI, London SE1 1FY, ⁴ National Primary Care Research and Development Centre, University of Manchester, Manchester M13 9PL, ⁵ Peninsula Medical School, Smeall Building, Exeter EX1 2LU

Objective To assess the robustness of patient responses to a new national survey of patient experience as a basis for providing financial incentives to doctors.

Design Analysis of the representativeness of the respondents to the GP Patient Survey compared with those who were sampled (5.5 million patients registered with 8273 general practices in England in January 2009) and with the general population. Analysis of non-response bias looked at the relation between practice response rates and scores on the survey. Analysis of the reliability of the survey estimated the proportion of the variance of practice scores attributable to true differences between practices.

Results The overall response rate was 38.2% (2.2 million responses), which is comparable to that in surveys using similar methodology in the UK. Men, young adults, and people living in deprived areas were under-represented among respondents. However, for questions related to pay for performance, there was no systematic association between response rates and questionnaire scores. Two questions which triggered payments to general practitioners were reliable measures of practice performance, with average practice-level reliability coefficients of 93.2% and 95.0%. Less than 3% and 0.5% of practices had fewer than the number of responses required to achieve conventional reliability levels of 90% and 70%. A change to the payment formula in 2009 resulted in an increase in the average impact of random variation in patient scores on payments to general practitioners compared with payments made in 2007 and 2008.

Conclusions There is little evidence to support the concern of some general practitioners that low response rates and selective non-response bias have led to systematic unfairness in payments attached to questionnaire scores. The study raises issues relating to the validity and reliability of payments based on patient surveys and provides lessons for the UK and for other countries considering the use of patient experience as part of pay for performance schemes.

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Articles

Effective, safe, and a good patient experience

Fiona Godlee
BMJ 2009 339: b4346. [Extract] [Full Text]

Using patient experience within pay for performance programmes

Chris Salisbury
BMJ 2009 339: b4224. [Extract] [Full Text]

Understanding why some ethnic minority patients evaluate medical care more negatively than white patients: a cross sectional analysis of a routine patient survey in English general practices

Nicola Mead and Martin Roland
BMJ 2009 339: b3450. [Abstract] [Full Text] [PDF]

This article has been cited by other articles:

• Salisbury, C. (2009). Using patient experience within pay for performance programmes. BMJ 339: b4224-b4224 [Full text]  

Rapid Responses:

Read all Rapid Responses

Why were some practices excluded and were these the ones appealing?

Mary Hawking
bmj.com, 12 Oct 2009 [Full text]

Online poll

Find out more on doc2doc >>