Questionnaire surveys: sources of biasBMJ 2013; 347 doi: https://doi.org/10.1136/bmj.f5265 (Published 30 August 2013) Cite this as: BMJ 2013;347:f5265
- Philip Sedgwick, reader in medical statistics and medical education
Researchers investigated the attitudes of patients about the use of placebo treatments in medical care. A telephone survey was used. Participants were residents in Northern California, aged 18-75 years, who had been seen by a primary care provider for a chronic health problem at least once in the previous six months.1
Participants were randomly selected using age stratified sampling (10 year age groups) from all residents in Northern California who met the inclusion criteria. An introductory letter describing the study’s aims was sent to 1800 residents inviting them to participate, with the opportunity to opt out of the study. Those who did not opt out were telephoned and asked to complete the survey by phone. Sample members were excluded if they had dementia or could not participate in a telephone interview because of communication barriers (hearing, language). Of the 1800 residents sampled, 1598 were reachable by mail and potentially eligible to participate. Of these, 853 completed the telephone interview, giving a response rate of 53.4% (853/1598).
The researchers concluded that most patients in the survey seemed favourable to the idea of placebo treatments, and that they valued honesty and transparency in this context. Respondents suggested that doctors should consider engaging with patients to discuss their values and attitudes about the appropriateness of using treatments aimed at promoting placebo responses.
Which of the following types of bias, if any, would the above questionnaire survey have been prone to?
a) Attrition bias
b) Non-response bias
c) Response bias
d) Selection bias
e) Volunteer bias
The questionnaire survey would have been prone to non-response bias (b), response bias (c), selection bias (d), and volunteer bias (e), but not attrition bias (a).
Non-response bias would have occurred if some people did not respond to the survey and they differed from the responders in their sociodemography, behaviour, or attitudes. The above survey was liable to non-response bias because not all of the sample members who were eligible to participate responded, there being a response rate of 53.4% (b is true). It is of interest why some of the original sample did not respond. However, any differences between the non-responders and responders may be difficult to quantify because limited information, if any, was available for those who did not participate. It would have been unethical to use previously collected information in patient databases, for example, about the non-responders, including their demographics, because they have not consented to be part of the study in any capacity. If non-response bias existed it would have threatened the external validity of the survey—that is, the extent to which the survey results could be generalised to the population.
Non-response bias should not be confused with response bias. Response bias would have occurred if there was systematic distortion in the way respondents answered questions. The survey was prone to response bias (c is true); it was possible the sample members responded about the use of placebo treatments in medical care in a way they perceived was of interest to the researchers. Response bias is a particular problem in questionnaire surveys that investigate socially unacceptable or embarrassing behaviours, such as alcohol consumption or drug taking.
Response bias is one of a group of biases collectively known as ascertainment bias, sometimes referred to as detection bias. Ascertainment bias is the systematic distortion of the assessment of outcome measures by the investigators or study participants. This group of biases is a particular problem in clinical trials when the researchers or trial participants are aware of the treatment allocation.2
Volunteer bias would have occurred if there was a systematic difference between those residents who volunteered to be part of the study (completed the survey) and the population. Although a random sample of residents from Northern California was originally selected, not all of them volunteered. The volunteers would be expected to differ from the original sample, and therefore the population, in their sociodemography, behaviour, and attitudes. Therefore, the above survey would have been prone to volunteer bias (e is true). However, it may be difficult to quantify the differences between the volunteers and the population. If volunteer bias existed it would have threatened the external validity of the survey—that is, the extent to which the survey results can be generalised to the population.
Non-response bias and volunteer bias are often confused. Non-response bias considers any differences between the non-responders and responders originally selected for the sample, whereas volunteer bias focuses on the differences between the volunteers and the population. Both will lead to a similar bias; in particular, the potential for the data collected not to be representative of the population but only those who choose to participate. It has been reported that, in general, those who participate in studies are more educated, come from a higher social class, and are more sociable than those who do not participate.
Selection bias would have occurred if there had been a systematic difference between those patients selected for the survey and those who were not, resulting in a sample that was not representative of the population. The original sample of 1800 adults was not a random sample from the theoretical infinite population but a sample of the eligible residents in northern California. The extent to which the residents of northern California were representative of the patient population was not clear. Therefore, because not all members of the population were eligible for selection, the above questionnaire was prone to selection bias (d is true). However, if the residents of northern California were representative of the population, because a random sample of these residents was selected, the original sample would not be prone to selection bias and d would be false.
Attrition bias occurs in longitudinal studies, such as cohort studies and clinical trials, when people are lost to follow-up in a non-random manner. Bias occurs if those lost to follow-up differ in some systematic way from those not lost to follow-up. It is a problem if the characteristics of those people who are lost to follow-up or the reasons for attrition are associated with the outcome measure(s). The questionnaire survey was a cross sectional study, with respondents surveyed at one point in time, so attrition bias did not occur (a is false).
Cite this as: BMJ 2013;347:f5265
Competing interests: None declared.