BMJ  2004;328:1312-1315 (29 May), doi:10.1136/bmj.328.7451.1312

Education and debate

Hands-on guide to questionnaire research

Selecting, designing, and developing your questionnaire

¹ Department of Primary Care and Population Sciences, University College London, Archway Campus, London N19 5LW

Anybody can write down a list of questions and photocopy it, but producing worthwhile and generalisable data from questionnaires needs careful planning and imaginative design

The great popularity with questionnaires is they provide a "quick fix" for research methodology. No single method has been so abused.¹

Questionnaires offer an objective means of collecting information about people's knowledge, beliefs, attitudes, and behaviour.^{2 3} Do our patients like our opening hours? What do teenagers think of a local antidrugs campaign and has it changed their attitudes? Why don't doctors use computers to their maximum potential? Questionnaires can be used as the sole research instrument (such as in a cross sectional survey) or within clinical trials or epidemiological studies.

Randomised trials are subject to strict reporting criteria,⁴ but there is no comparable framework for questionnaire research. Hence, despite a wealth of detailed guidance in the specialist literature,^{1-3 5 w1-w8} elementary methodological errors are common.¹ Inappropriate instruments and lack of rigour inevitably lead to poor quality data, misleading conclusions, and woolly recommendations.^w8 In this series we aim to present a practical guide that will enable research teams to do questionnaire research that is well designed, well managed, and non-discriminatory and which contributes to a generalisable evidence base. We start with selecting and designing the questionnaire.

What information are you trying to collect?

You and your co-researchers may have different assumptions about precisely what information you would like your study to generate. A formal scoping exercise will ensure that you clarify goals and if necessary reach an agreed compromise. It will also flag up potential practical problems—for example, how long the questionnaire will be and how it might be administered.

As a rule of thumb, if you are not familiar enough with the research area or with a particular population subgroup to predict the range of possible responses, and especially if such details are not available in the literature, you should first use a qualitative approach (such as focus groups) to explore the territory and map key areas for further study.⁶

Is a questionnaire appropriate?

People often decide to use a questionnaire for research questions that need a different method. Sometimes, a questionnaire will be appropriate only if used within a mixed methodology study—for example, to extend and quantify the findings of an initial exploratory phase. Table A on bmj.com gives some real examples where questionnaires were used inappropriately.¹

Box 1: Pitfalls of designing your own questionnaire Natasha, a practice nurse, learns that staff at a local police station have a high incidence of health problems, which she believes are related to stress at work. She wants to test the relation between stress and health in these staff to inform the design of advice services. Natasha designs her own questionnaire. Had she completed a thorough literature search for validated measures, she would have found several high quality questionnaires that measure stress in public sector workers.8 Natasha's hard work produces only a second rate study that she is unable to get published.

Research participants must be able to give meaningful answers (with help from a professional interviewer if necessary). Particular physical, mental, social, and linguistic needs are covered in the third article of this series.⁷

Could you use an existing instrument?

Using a previously validated and published questionnaire will save you time and resources; you will be able to compare your own findings with those from other studies, you need only give outline details of the instrument when you write up your work, and you may find it easier to get published (box 1).

Increasingly, health services research uses standard questionnaires designed for producing data that can be compared across studies. For example, clinical trials routinely include measures of patients' knowledge about a disease,⁹ satisfaction with services,¹⁰ or health related quality of life.^{11-13 w3 w9} The validity (see below) of this approach depends on whether the type and range of closed responses reflects the full range of perceptions and feelings that people in all the different potential sampling frames might hold. Importantly, health status and quality of life instruments lose their validity when used beyond the context in which they were developed. ^{12 14 15 w3 w10-12}

If there is no "off the peg" questionnaire available, you will have to construct your own. Using one or more standard instruments alongside a short bespoke questionnaire could save you the need to develop and validate a long list of new items.

Is the questionnaire valid and reliable?

A valid questionnaire measures what it claims to measure. In reality, many fail to do this. For example, a self completion questionnaire that seeks to measure people's food intake may be invalid because it measures what they say they have eaten, not what they have actually eaten.¹⁶ Similarly, responses on questionnaires that ask general practitioners how they manage particular clinical conditions differ significantly from actual clinical practice.^w13 An instrument developed in a different time, country, or cultural context may not be a valid measure in the group you are studying. For example, the item "I often attend gay parties" may have been a valid measure of a person's sociability level in the 1950s, but the wording has a very different connotation today.

Reliable questionnaires yield consistent results from repeated samples and different researchers over time. Differences in results come from differences between participants, not from inconsistencies in how the items are understood or how different observers interpret the responses. A standardised questionnaire is one that is written and administered so all participants are asked the precisely the same questions in an identical format and responses recorded in a uniform manner. Standardising a measure increases its reliability.

Just because a questionnaire has been piloted on a few of your colleagues, used in previous studies, or published in a peer reviewed journal does not mean it is either valid or reliable. The detailed techniques for achieving validity, reliability, and standardisation are beyond the scope of this series. If you plan to develop or modify a questionnaire yourself, you must consult a specialist text on these issues.^{2 3}

How should you present your questions?

Questionnaire items may be open or closed ended and be presented in various formats (figure). Table B on bmj.com examines the pros and cons of the two approaches. Two words that are often used inappropriately in closed question stems are frequently and regularly. A poorly designed item might read, "I frequently engage in exercise," and offer a Likert scale giving responses from "strongly agree" through to "strongly disagree." But "frequently" implies frequency, so a frequency based rating scale (with options such as at least once a day, twice a week, and so on) would be more appropriate. "Regularly," on the other hand, implies a pattern. One person can regularly engage in exercise once a month whereas another person can regularly do so four times a week. Other weasel words to avoid in question stems include commonly, usually, many, some, and hardly ever.^{17 w14}

View larger version (70K): [in this window] [in a new window]   Examples of formats for presenting questionnaire items

 

Box 2: A closed ended design that produced misleading information Customer: I'd like to discontinue my mobile phone rental please. Company employee: That's fine, sir, but I need to complete a form for our records on why you've made that decision. Is it (a) you have moved to another network; (b) you've upgraded within our network; or (c) you can't afford the payments? Customer: It isn't any of those. I've just decided I don't want to own a mobile phone any more. It's more hassle than it's worth. Company employee: [after a pause] In that case, sir, I'll have to put you down as "can't afford the payments."

Closed ended designs enable researchers to produce aggregated data quickly, but the range of possible answers is set by the researchers not respondents, and the richness of potential responses is lower. Closed ended items often cause frustration, usually because researchers have not considered all potential responses (box 2).¹⁸

Ticking a particular box, or even saying yes, no, or maybe can make respondents want to explain their answer, and such free text annotations may add richly to the quantitative data. You should consider inserting a free text box at the end of the questionnaire (or even after particular items or sections). Note that participants need instructions (perhaps with examples) on how to complete free text items in the same way as they do for closed questions.

If you plan to use open ended questions or invite free text comments, you must plan in advance how you will analyse these data (drawing on the skills of a qualitative researcher if necessary).¹⁹ You must also build into the study design adequate time, skills, and resources for this analysis; otherwise you will waste participants' and researchers' time. If you do not have the time or expertise to analyse free text responses, do not invite any.

Some respondents (known as yea sayers) tend to agree with statements rather than disagree. For this reason, do not present your items so that strongly agree always links to the same broad attitude. For example, on a patient satisfaction scale, if one question is "my GP generally tries to help me out," another question should be phrased in the negative, such as "the receptionists are usually impolite."

Apart from questions, what else should you include?

A common error by people designing questionnaires for the first time is simply to hand out a list of the questions they want answered. Table C on bmj.com gives a checklist of other things to consider. It is particularly important to provide an introductory letter or information sheet for participants to take away after completing the questionnaire.

What should the questionnaire look like?

Researchers rarely spend sufficient time on the physical layout of their questionnaire, believing that the science lies in the content of the questions and not in such details as the font size or colour. Yet empirical studies have repeatedly shown that low response rates are often due to participants being unable to read or follow the questionnaire (box 3).^{3 w6} In general, questions should be short and to the point (around 12 words or less), but for issues of a sensitive and personal nature, short questions can be perceived as abrupt and threatening, and longer sentences are preferred.^w6

How should you select your sample?

Different sampling techniques will affect the questions you ask and how you administer your questionnaire (see table D on bmj.com). For more detailed advice on sampling, see Bowling²⁰ and Sapsford.³

If you are collecting quantitative data with a view to testing a hypothesis or assessing the prevalence of a disease or problem (for example, about intergroup differences in particular attitudes or health status), seek statistical advice on the minimum sample size.³

What approvals do you need before you start?

Unlike other methods, questionnaires require relatively little specialist equipment or materials, which means that inexperienced and unsupported researchers sometimes embark on questionnaire surveys without completing the necessary formalities. In the United Kingdom, a research study on NHS patients or staff must be:

• Formally approved by the relevant person in an organisation that is registered with the Department of Health as a research sponsor (typically, a research trust, university or college)²¹;
  
• Consistent with data protection law and logged on the organisation's data protection files (see next article in series)¹⁹
  
• Accordant with research governance frameworks²¹
  
• Approved by the appropriate research ethics committee (see below).
  

Box 3: Don't let layout let you down Meena, a general practice tutor, wanted to study her fellow general practitioners' attitudes to a new training scheme in her primary care trust. She constructed a series of questions, but when they were written down, they covered 10 pages, which Meena thought looked off putting. She reduced the font and spacing of her questionnaire, and printed it double sided, until it was only four sides in length. But many of her colleagues refused to complete it, telling her they found it too hard to read and work through. She returned the questionnaire to its original 10 page format, which made it easier and quicker to complete, and her response rate increased greatly.

Summary points Questionnaire studies often fail to produce high quality generalisable data When possible, use previously validated questionnaires Questions must be phrased appropriately for the target audience and information required Good explanations and design will improve response rates

In addition, if your questionnaire study is part of a formal academic course (for example, a dissertation), you must follow any additional regulations such as gaining written approval from your supervisor.

A study is unethical if it is scientifically unsound, causes undue offence or trauma, breaches confidentiality, or wastes people's time or money. Written approval from a local or multicentre NHS research ethics committee (more information at www.corec.org.uk) is essential but does not in itself make a study ethical. Those working in non-NHS institutions or undertaking research outside the NHS may need to submit an additional (non-NHS) ethical committee application to their own institution or research sponsor.

The committee will require details of the study design, copies of your questionnaire, and any accompanying information or covering letters. If the questionnaire is likely to cause distress, you should include a clear plan for providing support to both participants and researchers. Remember that just because you do not find a question offensive or distressing does not mean it will not upset others.⁶

Conclusion

As we have shown above, designing a questionnaire study that produces usable data is not as easy as it might seem. Awareness of the pitfalls is essential both when planning research and appraising published studies. Table E on bmj.com gives a critical appraisal checklist for evaluating questionnaire studies. In the following two articles we will discuss how to select a sample, pilot and administer a questionnaire, and analyse data and approaches for groups that are hard to research.

---

This is the first in a series of three articles on questionnaire research

References w1-w17, further illustrative examples, and checklists are on bmj.com

Susan Catt supplied additional references and feedback. We also thank Alicia O'Cathain, Jill Russell, Geoff Wong, Marcia Rigby, Sara Shaw, Fraser MacFarlane, and Will Callaghan for feedback on earlier versions. Numerous research students and conference delegates provided methodological questions and case examples of real life questionnaire research, which provided the inspiration and raw material for this series. We also thank the hundreds of research participants who over the years have contributed data and given feedback to our students and ourselves about the design, layout, and accessibility of instruments.

Contributors and sources: PMB and TG have taught research methods in a primary care setting for the past 13 years, specialising in practical approaches and using the experiences and concerns of researchers and participants as the basis of learning. This series of papers arose directly from questions asked about real questionnaire studies. To address these questions we explored a wide range of sources from the psychological and health services research literature.

Competing interests: None declared.

References

1. Gillham B. Developing a questionnaire (real world research). London: Continuum, 2000.
2. Oppenheim AN. Questionnaire design, interviewing and attitude measurement. London: Continuum, 1992.
3. Sapsford R. Survey research. London: Sage, 1999.
4. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001;134: 663-94.[Abstract/Free Full Text]
5. McColl E, Thomas R. The use and design of questionnaires. London: Royal College of General Practitioners, 2000.
6. Howitt D, Cramer D. First steps in research and statistics. London: Routledge, 2000.
7. Boynton PM, Wood GW, Greenhalgh T. Hands-on guide to questionnaire research: reaching beyond the white middle classes. BMJ (in press).
8. Widerszal-Bazyl M, CieSlak R. Monitoring psychosocial stress at work: development of the psychosocial working conditions questionnaire. Int J Occup Saf Ergon 2000;special issue: 59-70.
9. Bradley C. Handbook of psychology of diabetes. London: Psychology Press, 1994.
10. Howie JG, Heaney DJ, Maxwell M, Walker JJ. A comparison of a patient enablement instrument (PEI) against two established satisfaction scales as an outcome measure of primary care consultations. Fam Pract 1998;15: 165-71.[Abstract/Free Full Text]
11. Van Hook MP, Berkman B, Dunkle R. Assessment tools for general health care settings: PRIME-MD, OARS, and SF-36. Health Soc Work 1996;21: 230-4.[ISI][Medline]
12. Lohr KN. Assessing health status and quality-of-life instruments: attributes and review criteria. Qual Life Res 2002;11: 193-205.[CrossRef][ISI][Medline]
13. Garratt A, Schmidt L, Mackintosh A, Fitzpatrick R. Quality of life measurement: bibliographic study of patient assessed health outcome measures. BMJ 2002;324: 1417.[Abstract/Free Full Text]
14. Dijkers M. Measuring quality of life: methodological issues. Am J Phys Med Rehabil 1999;78: 286-300.[CrossRef][ISI][Medline]
15. Gilbody SM, House AO, Sheldon TA. Routinely administered questionnaires for depression and anxiety: systematic review. BMJ 2001;322: 406-9.[Abstract/Free Full Text]
16. Drewnowski A. Diet image: a new perspective on the food-frequency questionnaire. Nutr Rev 2001;59: 370-2.[ISI][Medline]
17. Schaeffer NC. Hardly ever or constantly? Group comparisons using vague quantifiers. Public Opin Q 2003;55: 395-423.
18. Houtkoop-Steenstra H. Interaction and the standardised survey interview: the living questionnaire. Cambridge: Cambridge University Press, 2000.
19. Boynton PM. Hands-on guide to questionnaire research: administering, analysing, and reporting your questionnaire. BMJ (in press).
20. Bowling A. Research methods in health: investigating health and health services. Buckingham: Open University Press, 2000.
21. Department of Health. Research governance framework for health and social care. London: Stationery Office, 2002. www.dh.gov.uk/assetRoot/04/01/47/57/04014757.pdf (accessed 30 Apr 2004).

(Accepted 17 March 2004)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

This article has been cited by other articles:

• Lapi, F., Vannacci, A., Moschini, M., Cipollini, F., Morsuillo, M., Gallo, E., Banchelli, G., Cecchi, E., Pirro, M. D., Giovannini, M. G., Cariglia, M. T., Gori, L., Firenzuoli, F., Mugelli, A. (2008). Use, Attitudes and Knowledge of Complementary and Alternative Drugs (CADs) Among Pregnant Women: a Preliminary Survey in Tuscany. Evid Based Complement Alternat Med 0: nen031v1-nen031 [Abstract] [Full text]  
• Mayland, C., Williams, E., Ellershaw, J. (2008). How well do current instruments using bereaved relatives' views evaluate care for dying patients?. Palliat Med 22: 133-144 [Abstract]  
• Rosen, T., Olsen, J. (2006). Invited Commentary: The Art of Making Questionnaires Better. Am J Epidemiol 164: 1145-1149 [Abstract] [Full text]  
• Schilling, L. M., Kozak, K., Lundahl, K., Dellavalle, R. P. (2006). Inaccessible Novel Questionnaires in Published Medical Research: Hidden Methods, Hidden Costs. Am J Epidemiol 164: 1141-1144 [Abstract] [Full text]  
• Aronson, J. K (2006). ;Open-access' publishing: first the evidence--then the verdict.. JRSM 99: 103-104 [Full text]  
• Boynton, P. M (2004). Administering, analysing, and reporting your questionnaire. BMJ 328: 1372-1375 [Full text]  

Rapid Responses:

Read all Rapid Responses

Questionnaire Research - personal experience

Dr.Sen V Kallumpuram
bmj.com, 3 Jun 2004 [Full text]

Even if the questions do offend…can we change a standardised questionnaire?

Elise C Maher
bmj.com, 8 Jul 2004 [Full text]