General practice

Comparison of patient questionnaire, medical record, and audio tape in assessment of health promotion in general practice consultations

^a Department of General Practice, University of Leicester, Leicester General Hospital, Leicester LE5 4PW, ^b Department of General Practice, University of Nottingham, Nottingham

Correspondence to: Dr Wilson.

Abstract

Objective: To determine what proportion of health promotion activities reported by the patient is recorded in the general practice notes and to compare these methods of assessing health promotion with audio tape analysis.
Design: Secondary analysis of data obtained in a controlled trial of differing appointment lengths. After each consultation the medical record was examined and the patient invited to completed a questionnaire. A subsample of consultations was audio taped.
Setting: Nottinghamshire. Subjects--16 general practitioners from 10 practices. This report includes 3324 consultations with patients aged >/=17, with data on measurement of blood pressure and advice about smoking and alcohol.
Results: Data from questionnaire and medical notes were available for 2281 consultations. Advice on smoking was recorded in the notes in 30.9% of cases in which a patient reported it (for alcohol and measurement of blood pressure, 44.4% and 82.7% of cases respectively). In 516 cases analysis of audio tape and review of records was performed. Advice on smoking was recorded in the patient's notes in 28.6% of cases in which it was detected on audio tape (for alcohol, 31.1% of cases). In 335 consultations data from audio tape and questionnaire were available. Advice on smoking was reported by patients in 73.9% of cases in which it was detected on audio tape (for alcohol, 75.0% of cases).
Conclusions: Review of the medical record is a reasonably accurate method of assessing measureent of blood pressure in the consultation but would lead to significant underestimation of advice about smoking and alcohol.

Introduction

The emphasis on health promotion in general practice has shifted to opportunistic activities in the consultation, encouraged by revised arrangements for payment¹ and increasing evidence that health checks are most useful for those at highest risk.^{2 3} The extent of health promotion in the consultation is difficult to measure, and we lack valid and reliable methods of doing so. Analysis of audio tape or video tape recordings is generally considered to be the most valid method but is extremely time consuming and requires thorough training of assessors to ensure reliability.⁴ Because of these difficulties health promotion in the consultation has also been assessed with review of medical records⁵ and the use of patient questionnaires⁶ or interviews.⁷

Practices differ enormously in the amount of preventive data entered in patients' records.⁵ This may be a result of differing levels of health promotion activity or different recording practices, or both. It has been suggested that when blood pressure or weight is measured the value is nearly always recorded, but no evidence exists to support this.⁸ The assumption underlying the revised health promotion payments is that health promotion activity can be assessed by examining the extent of data collection, although this has not been proved.

Patient questionnaires have been advocated as a method of auditing consulting behaviour,⁹ but few studies have used this approach to measure specific health promotion activities. The method is attractive as a research tool as it is not time consuming and avoids some of the problems of bias and reliability that may complicate review of records or interviewing patients. It is known, however, that much of what happens in the consultation is quickly forgotten¹⁰ and that patients are apt to produce answers that they believe will please their doctor.¹¹

In a study of the effect of the length of consultation health promotion activity was assessed with review of patient records, patient questionnaires, and audio tape analysis of a subsample. In this report we present data on the proportion of health promotion activity reported by patients that is recorded in the patients' notes and compare the sensitivity and specificity of patient questionnaires and review of records with the "gold standard" of audio tape analysis.

Methods

We recruited 16 general practitioners from 10 practices to a trial using 10 minute appointments in general practice, the details of which have been described elsewhere.¹² For each general practitioner, we compared surgery sessions booked at 10 minute intervals with those booked at the usual rate of 7.5 minutes or less. In each surgery session patients were given a questionnaire to complete after seeing the doctor, and the medical record was reviewed by a research assistant. At least two surgery sessions from each doctor were selected for audio taping, subject to the patient's consent. This selection was not random but was based on practical constraints such as the availability of equipment.

The health promotion questions to patients included the question, "Did the doctor talk to you about smoking or alcohol or take your blood pressure?" which required a yes/no answer. Only adults aged 35-65 were asked the question about blood pressure as at the time of the study this was the target group for screening.

The same rules were applied for the analysis of audio tape and review of the medical record. Any mention by a doctor of smoking, alcohol, diet, or exercise counted as positive. A topic raised by a patient but ignored by the doctor did not count. Entry of a blood pressure recorded in the notes counted as positive. On audio tape, measurement of blood pressure was determined by verbal content or the sound of the procedure itself. None of the participating doctors used a computer terminal during the consultation. Data collection took place from 1987 to 1989, before the introduction of payments for recording risk factors.

This report is restricted to consultation with adults (aged 17 and over). In all, 3324 consultations are included. Of these, 3257 (98%) of medical records were reviewed, and in 2329 (70.1%) cases a questionnaire was completed. The number of consultations for which audio taped data are available is 528 (16.2%).

RELIABILITY ESTIMATES

We estimated reliability in two ways. Firstly, a second research assistant reviewed a subsample of the cases. Secondly, a third assessor (AW) reviewed a sample from both the research assistants. Agreement was estimated with Cohen's (Kappa) statistic for categorical data.¹³

Seventy five records were examined by two or more researchers. Cohen's (Kappa) statistic was assessed for the three pairs of researchers. The levels of agreement between pairs ranged from 0.96 to 1 for blood pressure and from 0.89 to 1 for smoking and alcohol combined. Similarly, 44 audio taped consultations were analysed by more than one observer. The levels of agreement ranged from 0.79 to 0.94 for blood pressure and from 0.84 to 0.97 for alcohol and smoking combined. Reliable estimates for smoking and alcohol cannot be given separately because of the small number of positive entries. (Kappa) Coefficients above 0.60 indicate "substantial" agreement and above 0.80 "near perfect" agreement.¹³

Results

NOTES VERSUS QUESTIONNAIRE

For 2281 consultations a patient questionnaire had been completed and the medical record reviewed. The number of missing values varied between questions. Table I shows the results of smoking, alcohol, and blood pressure. The denominator for each item differs as only those cases in which the respondent answered the relevant question are included. Those who did not declare their smoking habit have been excluded. This also accounts for differences between denominators in tables II and III. Every health promotion topic was reported more often by patients than it was recorded in the notes. A record both in the medical notes and in the patient's questionnaire of advice having been given occurred most often for the discussion of smoking with smokers (44.4%) and least often for the discussion of alcohol (30.9%). In over 80% of cases in which a patient reported assessment of blood pressure a result was entered in the notes.

TABLE I--Comparison of reporting of health promotion by patients completing a questionnaire compared
with recording in medical notes
     -------------------------------------------------------------------------------------------------
                                                   No (%) of patients           Percentage of
                                             ------------------------------   patients reporting
                            No of patients   Reporting        Advice or          advice whose
                               answering      advice or        procedure          notes also
Health promotion activity      question      procedure    recorded in notes      had a record
------------------------------------------------------------------------------------------------------
Smoking:
  All respondents                 2105        329 (15.6)      154 (7.3)          43.2 (142/329)
  Declared smokers                 616        151 (24.5)       71 (11.5)         44.4 (67/151)
  Declared non-smokers            1471        174 (11.8)       82 (5.6)          42.5 (74/174)
Alcohol                           2222        139 (6.3)        48 (2.2)          30.9 (43/139)
Blood pressure*                   1024        307 (30.0)      265 (25.9)         82.7 (254/307)
------------------------------------------------------------------------------------------------------
* Asked only of patients aged 35-65.

TABLE II--Comparison of recording of health promotion in notes compared with detection on audio tape
     -----------------------------------------------------------------------------------------------------------------
                                              No (%) of patients
                                      ----------------------------------
                     No of patients      Advice or      Record of advice   Sensitivity* of      % Of false
Health promotion       answering      procedure heard     or procedure     notes compared    positive entries
activity                question       on audio tape        in notes       with audio tape       in notes
     -----------------------------------------------------------------------------------------------------------------
Smoking:
  All respondents         516            45 (8.7)           19 (3.7)          31.1 (14/45)      1.1 (5/471)
  Declared
  smokers                  92            11 (12.0)           5 (5.4)          45.5 (5/11)         0 (0/81)
  Declared
  non-smokers             218            13 (6.0)            7 (3.2)          30.8 (4/13)       1.5 (3/205)
Alcohol                   516            14 (2.7)            6 (1.2)          28.6 (4/14)       0.4 (2/502)
Blood pressure            516           120 (23.3)         118 (22.9)         83.3 (100/120)    4.5 (18/396)
----------------------------------------------------------------------------------------------------------------------
* As a percentage.

TABLE III--Comparison of reporting of health promotion by patients compared with detection on audio tape
     ---------------------------------------------------------------------------------------------------------
                                            No (%) of patients
                                      -----------------------------
                                      Advice or                       Sensitivity* of    % Of false
                     No of patients   procedure                       questionnaire,      positive
Health promotion       answering       heard on    Reporting advice    compared with     answers in
activity                question      audio tape     or procedure       audio tape      questionnaire
     ---------------------------------------------------------------------------------------------------------
Smoking:
  All respondents         313          23 (7.3)        35 (11.2)         73.9 (17/23)    6.2 (18/290)
  Declared
  smokers                  97         11 (11.3)        18 (18.6)         81.8 (9/11)    10.5 (9/86)
  Declared
  non-smokers             214          11 (5.1)        17 (7.9)          72.7 (8/11)     4.4 (9/203)
Alcohol                   335          8 (2.4)         15 (4.5)          75.0 (6/8)      2.8 (9/327)
Blood pressure            151         30 (19.9)        42 (27.8)        100.0 (30/30)    9.9 (12/121)
--------------------------------------------------------------------------------------------------------------
* As a percentage.

NOTES VERSUS AUDIO TAPE

In 516 consultations an audio tape analysis and a review of records were performed. Table II shows how often health promotion items were detected by both methods. The "sensitivity" of the notes is the proportion of cases detected on audio tape that are accompanied by an entry in the notes. For smoking and alcohol this ranged from 45.5% for the discussion of smoking with smokers to 28.6% for the discussion of alcohol. False positive entries are entries in the notes that were not detected on audio tape. The rate of false positive entries ranged from 0.4% for the discussion of alcohol to 1.5% for the discussion of smoking with non-smokers. The sensitivity of the notes for the measurement of blood pressure (83.3%) was much higher than that for discussion of smoking or alcohol.

QUESTIONNAIRE VERSUS AUDIO TAPE

Results from a review of the medical record and from audio tape analysis were available for 335 consultations (table III). All items were reported by the patient more often than they were detected on the audio tape. This discrepancy was reflected in the high rate of false positive answers in the questionnaire--that is, items reported by the patient but not detected on audio tape--ranging from 4.4% for the discussion of smoking with non-smokers to 10.5% for the discussion of smoking with smokers. The sensitivity of the questionnaire was high for most items and reached 100% for the assessment of blood pressure.

None of the results differed between the consultations booked at 10 minute intervals and those that took place at the doctors' usual rates.

Discussion

We have found that the proportions of health promotion activities both reported by the patient and recorded in the notes vary depending on the health issue examined. On the larger dataset we have shown that for smoking about two fifths of activities reported by the patient are recorded in the notes and for alcohol about a third. In over 80% of cases in which the patient reported a measurement of blood pressure a reading was found in the medical record.

AUDIO TAPE AS A GOLD STANDARD

A problem with assessing any method of estimating the extent of health promotion in general practice is the lack of a gold standard that accurately measures what happened in the consultation. For the subset of consultations that were audio taped we have assumed that analysis of these tapes is the closest we can get to such a standard. Clearly some errors exist in such an assumption. However, the high level of interrater reliability on audio tape analysis and the small percentage of cases in which a health promotion entry existed in the medical records but was not detected on audio tape suggests that the errors are small. It is reasonable to assume that the rates of false positive entries of 1% for smoking and 0.5% for alcohol reflect cases in which the researcher did not identify a health promotion component. These rates may be slight overestimates as in some cases an entry at the time of the consultation may have been based on a doctor's prior knowledge rather than a component of the consultation. The higher rate of 5% for blood pressure reflects the difficulty in assessing this procedure with an audio tape.

FALSE NEGATIVE ENTRIES IN RECORDS AND PATIENT QUESTIONNAIRE

Of discussions of smoking heard on audio tape, a third had been entered in the medical records and three quarters had been reported by patients. Although the numbers are too small for statistical testing, discussion of smoking with smokers seems more likely to be both reported and recorded than discussion of this topic with non-smokers. This finding is compatible with the findings of Pill et al that the recording of advice having been given occurred in those cases in which the patient was at highest risk.⁷ This bias would lead to an overestimation of smoking prevalence if data collected were extrapolated to the practice population. For the audio taped subset the recording and reporting of advice having been given occurred at closely similar levels for alcohol and smoking, although in the whole dataset this recording reported by patients was lower for alcohol than smoking (31% and 43% respectively). Similar ascertainment biases probably apply as for smoking.

FALSE POSITIVE ENTRIES IN QUESTIONNAIRE

A surprising number of cases existed in which the patient reported health promotion in the consultation but none was detected on the audio tape. This discrepancy may be due in part to observer error, but as discussed previously, this effect is probably small. Another explanation is that patients wanted to give answers that would please their doctor, but this does not explain differences in the rates of false positive entries between smoking and alcohol. The likeliest explanation is that some patients found it difficult to remember whether advice from their doctor was given at the immediately preceding consultation or on an earlier occasion. The high rate of false positive entries in questionnaires for the measurement of blood pressure may be due in part to confusion about the nature of the procedure.

CONCLUSIONS

Overall, the results suggest that review of the medical record is a reasonably accurate method of assessing whether blood pressure has been measured in the consultation. For smoking and alcohol, however, this method would significantly underestimate the number of occasions in which advice is given, and this could lead to bias if results are extrapolated to the practice population. After the introduction of the new system of payment for health promotion¹ recording practices will probably have changed, and computer prompts are now used increasingly in consultations.⁴ Further studies are needed to assess the effects of these changes on recording practices. The problem of using the medical record as a source of information about any aspect of performance has recently been emphasised.¹⁴

A questionnaire administered after the consultation has an acceptable sensitivity in estimating whether advice about smoking and alcohol has been given but is less useful for blood pressure. Those at risk may be more likely to report that they received advice on smoking or drinking-- when it has been given--and to remember falsely a discussion of these topics. Such errors can be estimated and are likely to differ depending on the questionnaire and the context of its administration. Despite these reservations, a patient questionnaire may be the most feasible method of assessing in large scale research or audit studies whether advice on lifestyle--for example, smoking or drinking--has been given.

We thank participating doctors and practice staff for their collaboration and Leslie Hayes and Jacquie Cooney for collection of additional data. The study was funded by the Health Promotion Research Trust.

1. Department of Health. Statement of fees and allowances payable to general medical practitioners. London: DoH, 1992.
2. Family Heart Study Group. Randomised controlled trial evaluating cardiovascular screening and intervention in general practice: principal results of British family heart study. BMJ 1994;308:313-20. [Abstract/Free Full Text]
3. Imperial Cancer Research Fund OXCHECK Study Group. Effectiveness of health checks conducted by nurses in primary care: results of the OXCHECK study after one year. BMJ 1994;308:308-12. [Abstract/Free Full Text]
4. Pringle M, Robins S, Brown G. TIMER: a new objective measure of consultation content and its application to computer assisted consultations BMJ 1986;293:20-22.
5. Maitland JM, Reid J, Taylor RJ. Two stage audit of cerebrovascular and coronary heart disease risk factor recording: the effect of case finding and screening programmes. Br J Gen Pract 1991;41:144-6. [Medline]
6. Silagy C, Muir J, Coulter A, Thorogood M, Yudkin P, Roe L. Lifestyle advice in general practice: rates recalled by patients. BMJ 1992;305:871-4.
7. Pill RM, Jones-Elwyn G, Stott NCH. Opportunistic health promotion quantity or quality? J R Coll Gen Pract 1989;39:196-200.
8. Mant D, McKinlay C, Fuller A, Randall T, Fullard EM, Muir J. Three year follow up of patients with raised blood pressure identified at health checks in general practice. BMJ 1989;298:1360-2.
9. Baker R. Development of a questionnaire to assess patients' satisfaction with consultations in general practice. Br J Gen Pract 1990;40:487-90. [Medline]
10. Ley P, Bradshaw PW, Eaves D, Walker CM. A method for increasing patients' recall of information presented by doctors. Psychol Med 1973;3:217-20. [Medline]
11. Fitzpatrick R. Surveys of patient satisfaction: II--Designing a questionnaire and conducting a survey. BMJ 1991;302:1129-32.
12. Wilson A, McDonald P, Hayes L, Cooney J. Health promotion in the general practice consultation: a minute makes a difference. BMJ 1992;304:227-30.
13. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-74. [Medline]
14. Rethans JJ, Martin E, Metsemakers J. To what extent do clinical notes by general practitioners reflect actual medical performance? A study using simulated patients. Br J Gen Pract 1994;44:153-6. [Medline]

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

This article has been cited by other articles:

• McCullough, A., Fisher, M., Goldstein, A. O., Kramer, K. D., Ripley-Moffitt, C. (2009). Smoking As a Vital Sign: Prompts to Ask and Assess Increase Cessation Counseling. J Am Board Fam Med 22: 625-632 [Abstract] [Full text]  
• McKee, S. A., Falba, T., O'Malley, S. S., Sindelar, J., O'Connor, P. G. (2007). Smoking Status as a Clinical Indicator for Alcohol Misuse in US Adults. Arch Intern Med 167: 716-721 [Abstract] [Full text]  
• Winickoff, J. P., Tanski, S. E., McMillen, R. C., Hipple, B. J., Friebely, J., Healey, E. A. (2006). A National Survey of the Acceptability of Quitlines to Help Parents Quit Smoking. Pediatrics 117: e695-e700 [Abstract] [Full text]  
• Johansson, K., Bendtsen, P., Akerlind, I. (2005). Advice to patients in Swedish primary care regarding alcohol and other lifestyle habits: how patients report the actions of GPs in relation to their own expectations and satisfaction with the consultation. Eur J Public Health 15: 615-620 [Abstract] [Full text]  
• Wilson, A, Hippisley-Cox, J, Coupland, C, Coleman, T, Britton, J, Barrett, S (2005). Smoking cessation treatment in primary care: prospective cohort study. Tobacco Control 14: 242-246 [Abstract] [Full text]  
• Winickoff, J. P., Tanski, S. E., McMillen, R. C., Klein, J. D., Rigotti, N. A., Weitzman, M. (2005). Child Health Care Clinicians' Use of Medications to Help Parents Quit Smoking: A National Parent Survey. Pediatrics 115: 1013-1017 [Abstract] [Full text]  
• Spies, T. H., Mokkink, H. G. A., De Vries Robbe, P. F., Grol, R. P. T. M. (2004). Which data source in clinical performance assessment? A pilot study comparing self-recording with patient records and observation. Int J Qual Health Care 16: 65-72 [Abstract] [Full text]  
• McCurry, N., Thompson, K., Parahoo, K., O'Doherty, E., Doherty, A.-M. (2002). Pregnant women's perception of the implementation of smoking cessation advice. Health Education Journal 61: 20-31 [Abstract]  
• Nicholson, J. M, Hennrikus, D. J, Lando, H. A, McCarty, M. C, Vessey, J. (2000). Patient recall versus physician documentation in report of smoking cessation counselling performed in the inpatient setting. Tobacco Control 9: 382-388 [Abstract] [Full text]  
• Shakeshaft, A. P., Bowman, J. A., Sanson-Fisher, R. W. (1999). A COMPARISON OF TWO RETROSPECTIVE MEASURES OF WEEKLY ALCOHOL CONSUMPTION: DIARY AND QUANTITY/FREQUENCY INDEX. Alcohol Alcohol 34: 636-645 [Abstract] [Full text]  
• HOBBS, M., WARD, J., D'ESTE, K., DONNELLY, N. (1998). Consultations for cervical smears in general practice: a missed opportunity for smoking cessation advice?. Tobacco Control 7: 193a-194 [Full text]  
• McBride, P. E., Plane, M. B., Underbakke, G., Brown, R. L., Solberg, L. I. (1997). Smoking Screening and Management in Primary Care Practices. Arch Fam Med 6: 165-172 [Abstract]  
• Brook, R. H., McGlynn, E. A., Cleary, P. D. (1996). Measuring Quality of Care- Part Two of Six. NEJM 335: 966-970 [Full text]