Comparison of patient questionnaire, medical record, and audio tape in assessment of health promotion in general practice consultationsBMJ 1994; 309 doi: http://dx.doi.org/10.1136/bmj.309.6967.1483 (Published 03 December 1994) Cite this as: BMJ 1994;309:1483
- a Department of General Practice, University of Leicester, Leicester General Hospital, Leicester LE5 4PW
- b Department of General Practice, University of Nottingham, Nottingham
- Correspondence to: Dr Wilson.
- Accepted 4 November 1994
Objective: To determine what proportion of health promotion activities reported by the patient is recorded in the general practice notes and to compare these methods of assessing health promotion with audio tape analysis.
Design: Secondary analysis of data obtained in a controlled trial of differing appointment lengths. After each consultation the medical record was examined and the patient invited to completed a questionnaire. A subsample of consultations was audio taped.
Setting: Nottinghamshire. Subjects—16 general practitioners from 10 practices. This report includes 3324 consultations with patients aged >/=17, with data on measurement of blood pressure and advice about smoking and alcohol.
Results: Data from questionnaire and medical notes were available for 2281 consultations. Advice on smoking was recorded in the notes in 30.9% of cases in which a patient reported it (for alcohol and measurement of blood pressure, 44.4% and 82.7% of cases respectively). In 516 cases analysis of audio tape and review of records was performed. Advice on smoking was recorded in the patient's notes in 28.6% of cases in which it was detected on audio tape (for alcohol, 31.1% of cases). In 335 consultations data from audio tape and questionnaire were available. Advice on smoking was reported by patients in 73.9% of cases in which it was detected on audio tape (for alcohol, 75.0% of cases).
Conclusions: Review of the medical record is a reasonably accurate method of assessing measureent of blood pressure in the consultation but would lead to significant underestimation of advice about smoking and alcohol.
Assessment of health promotion in general practice is needed for research, audit, and contractual reasons
The accepted “gold standard” for measurement is audio tape or video tape analysis; less labour intensive methods include review of records and patient questionnaires
Medical records seriously underestimate the occasions on which advice on lifestyle is given but are reasonably accurate for the measurement of blood pressure
Questionnaires completed by patients after consultation overestimate health promotion activity
Changes in ascertainment of lifestyle information since the new contract may reflect changing recording practices rather than increased health promotion activity
The emphasis on health promotion in general practice has shifted to opportunistic activities in the consultation, encouraged by revised arrangements for payment1 and increasing evidence that health checks are most useful for those at highest risk.2 3 The extent of health promotion in the consultation is difficult to measure, and we lack valid and reliable methods of doing so. Analysis of audio tape or video tape recordings is generally considered to be the most valid method but is extremely time consuming and requires thorough training of assessors to ensure reliability.4 Because of these difficulties health promotion in the consultation has also been assessed with review of medical records5 and the use of patient questionnaires6 or interviews.7
Practices differ enormously in the amount of preventive data entered in patients' records.5 This may be a result of differing levels of health promotion activity or different recording practices, or both. It has been suggested that when blood pressure or weight is measured the value is nearly always recorded, but no evidence exists to support this.8 The assumption underlying the revised health promotion payments is that health promotion activity can be assessed by examining the extent of data collection, although this has not been proved.
Patient questionnaires have been advocated as a method of auditing consulting behaviour,9 but few studies have used this approach to measure specific health promotion activities. The method is attractive as a research tool as it is not time consuming and avoids some of the problems of bias and reliability that may complicate review of records or interviewing patients. It is known, however, that much of what happens in the consultation is quickly forgotten10 and that patients are apt to produce answers that they believe will please their doctor.11
In a study of the effect of the length of consultation health promotion activity was assessed with review of patient records, patient questionnaires, and audio tape analysis of a subsample. In this report we present data on the proportion of health promotion activity reported by patients that is recorded in the patients' notes and compare the sensitivity and specificity of patient questionnaires and review of records with the “gold standard” of audio tape analysis.
We recruited 16 general practitioners from 10 practices to a trial using 10 minute appointments in general practice, the details of which have been described elsewhere.12 For each general practitioner, we compared surgery sessions booked at 10 minute intervals with those booked at the usual rate of 7.5 minutes or less. In each surgery session patients were given a questionnaire to complete after seeing the doctor, and the medical record was reviewed by a research assistant. At least two surgery sessions from each doctor were selected for audio taping, subject to the patient's consent. This selection was not random but was based on practical constraints such as the availability of equipment.
The health promotion questions to patients included the question, “Did the doctor talk to you about smoking or alcohol or take your blood pressure?” which required a yes/no answer. Only adults aged 35-65 were asked the question about blood pressure as at the time of the study this was the target group for screening.
The same rules were applied for the analysis of audio tape and review of the medical record. Any mention by a doctor of smoking, alcohol, diet, or exercise counted as positive. A topic raised by a patient but ignored by the doctor did not count. Entry of a blood pressure recorded in the notes counted as positive. On audio tape, measurement of blood pressure was determined by verbal content or the sound of the procedure itself. None of the participating doctors used a computer terminal during the consultation. Data collection took place from 1987 to 1989, before the introduction of payments for recording risk factors.
This report is restricted to consultation with adults (aged 17 and over). In all, 3324 consultations are included. Of these, 3257 (98%) of medical records were reviewed, and in 2329 (70.1%) cases a questionnaire was completed. The number of consultations for which audio taped data are available is 528 (16.2%).
We estimated reliability in two ways. Firstly, a second research assistant reviewed a subsample of the cases. Secondly, a third assessor (AW) reviewed a sample from both the research assistants. Agreement was estimated with Cohen's (Kappa) statistic for categorical data.13
Seventy five records were examined by two or more researchers. Cohen's (Kappa) statistic was assessed for the three pairs of researchers. The levels of agreement between pairs ranged from 0.96 to 1 for blood pressure and from 0.89 to 1 for smoking and alcohol combined. Similarly, 44 audio taped consultations were analysed by more than one observer. The levels of agreement ranged from 0.79 to 0.94 for blood pressure and from 0.84 to 0.97 for alcohol and smoking combined. Reliable estimates for smoking and alcohol cannot be given separately because of the small number of positive entries. (Kappa) Coefficients above 0.60 indicate “substantial” agreement and above 0.80 “near perfect” agreement.13
NOTES VERSUS QUESTIONNAIRE
For 2281 consultations a patient questionnaire had been completed and the medical record reviewed. The number of missing values varied between questions. Table I shows the results of smoking, alcohol, and blood pressure. The denominator for each item differs as only those cases in which the respondent answered the relevant question are included. Those who did not declare their smoking habit have been excluded. This also accounts for differences between denominators in tables II and III. Every health promotion topic was reported more often by patients than it was recorded in the notes. A record both in the medical notes and in the patient's questionnaire of advice having been given occurred most often for the discussion of smoking with smokers (44.4%) and least often for the discussion of alcohol (30.9%). In over 80% of cases in which a patient reported assessment of blood pressure a result was entered in the notes.
NOTES VERSUS AUDIO TAPE
In 516 consultations an audio tape analysis and a review of records were performed. Table II shows how often health promotion items were detected by both methods. The “sensitivity” of the notes is the proportion of cases detected on audio tape that are accompanied by an entry in the notes. For smoking and alcohol this ranged from 45.5% for the discussion of smoking with smokers to 28.6% for the discussion of alcohol. False positive entries are entries in the notes that were not detected on audio tape. The rate of false positive entries ranged from 0.4% for the discussion of alcohol to 1.5% for the discussion of smoking with non-smokers. The sensitivity of the notes for the measurement of blood pressure (83.3%) was much higher than that for discussion of smoking or alcohol.
QUESTIONNAIRE VERSUS AUDIO TAPE
Results from a review of the medical record and from audio tape analysis were available for 335 consultations (table III). All items were reported by the patient more often than they were detected on the audio tape. This discrepancy was reflected in the high rate of false positive answers in the questionnaire—that is, items reported by the patient but not detected on audio tape—ranging from 4.4% for the discussion of smoking with non-smokers to 10.5% for the discussion of smoking with smokers. The sensitivity of the questionnaire was high for most items and reached 100% for the assessment of blood pressure.
None of the results differed between the consultations booked at 10 minute intervals and those that took place at the doctors' usual rates.
We have found that the proportions of health promotion activities both reported by the patient and recorded in the notes vary depending on the health issue examined. On the larger dataset we have shown that for smoking about two fifths of activities reported by the patient are recorded in the notes and for alcohol about a third. In over 80% of cases in which the patient reported a measurement of blood pressure a reading was found in the medical record.
AUDIO TAPE AS A GOLD STANDARD
A problem with assessing any method of estimating the extent of health promotion in general practice is the lack of a gold standard that accurately measures what happened in the consultation. For the subset of consultations that were audio taped we have assumed that analysis of these tapes is the closest we can get to such a standard. Clearly some errors exist in such an assumption. However, the high level of interrater reliability on audio tape analysis and the small percentage of cases in which a health promotion entry existed in the medical records but was not detected on audio tape suggests that the errors are small. It is reasonable to assume that the rates of false positive entries of 1% for smoking and 0.5% for alcohol reflect cases in which the researcher did not identify a health promotion component. These rates may be slight overestimates as in some cases an entry at the time of the consultation may have been based on a doctor's prior knowledge rather than a component of the consultation. The higher rate of 5% for blood pressure reflects the difficulty in assessing this procedure with an audio tape.
FALSE NEGATIVE ENTRIES IN RECORDS AND PATIENT QUESTIONNAIRE
Of discussions of smoking heard on audio tape, a third had been entered in the medical records and three quarters had been reported by patients. Although the numbers are too small for statistical testing, discussion of smoking with smokers seems more likely to be both reported and recorded than discussion of this topic with non-smokers. This finding is compatible with the findings of Pill et al that the recording of advice having been given occurred in those cases in which the patient was at highest risk.7 This bias would lead to an overestimation of smoking prevalence if data collected were extrapolated to the practice population. For the audio taped subset the recording and reporting of advice having been given occurred at closely similar levels for alcohol and smoking, although in the whole dataset this recording reported by patients was lower for alcohol than smoking (31% and 43% respectively). Similar ascertainment biases probably apply as for smoking.
FALSE POSITIVE ENTRIES IN QUESTIONNAIRE
A surprising number of cases existed in which the patient reported health promotion in the consultation but none was detected on the audio tape. This discrepancy may be due in part to observer error, but as discussed previously, this effect is probably small. Another explanation is that patients wanted to give answers that would please their doctor, but this does not explain differences in the rates of false positive entries between smoking and alcohol. The likeliest explanation is that some patients found it difficult to remember whether advice from their doctor was given at the immediately preceding consultation or on an earlier occasion. The high rate of false positive entries in questionnaires for the measurement of blood pressure may be due in part to confusion about the nature of the procedure.
Overall, the results suggest that review of the medical record is a reasonably accurate method of assessing whether blood pressure has been measured in the consultation. For smoking and alcohol, however, this method would significantly underestimate the number of occasions in which advice is given, and this could lead to bias if results are extrapolated to the practice population. After the introduction of the new system of payment for health promotion1 recording practices will probably have changed, and computer prompts are now used increasingly in consultations.4 Further studies are needed to assess the effects of these changes on recording practices. The problem of using the medical record as a source of information about any aspect of performance has recently been emphasised.14
A questionnaire administered after the consultation has an acceptable sensitivity in estimating whether advice about smoking and alcohol has been given but is less useful for blood pressure. Those at risk may be more likely to report that they received advice on smoking or drinking— when it has been given—and to remember falsely a discussion of these topics. Such errors can be estimated and are likely to differ depending on the questionnaire and the context of its administration. Despite these reservations, a patient questionnaire may be the most feasible method of assessing in large scale research or audit studies whether advice on lifestyle—for example, smoking or drinking—has been given.
We thank participating doctors and practice staff for their collaboration and Leslie Hayes and Jacquie Cooney for collection of additional data. The study was funded by the Health Promotion Research Trust.