Administering, analysing, and reporting your questionnaireBMJ 2004; 328 doi: https://doi.org/10.1136/bmj.328.7452.1372 (Published 03 June 2004) Cite this as: BMJ 2004;328:1372
Documenting data on refusals
Phuong is leading a study on new parents’ experiences of maternity services. She invites people to participate, and notes the age, sex, ethnicity, and reasons given for refusing to join the study. Most commonly, participants refuse because they are too busy, not interested in the research, or dislike the subject matter of the questionnaire. By keeping tabs on participants who refuse, Phuong can check that no particular subgroup of people is being excluded, and she can later formally compare the characteristics of responders with non-responders. An example of the form used for collecting exclusion/refusal data in Phuong’s study is shown below:
Reason given for non-participation
Ways of keeping quantitative data clean10
1. Be as careful and accurate as possible when entering data into a database.
2. Take breaks. Fatigue causes mistakes.
3. Double check your questionnaire against the data you have entered. You can either do this with every questionnaire, or by picking a random sample and checking how accurately the data in the database matches the answers on the questionnaire.
4. If possible, get a colleague to work with you (eg read out answers from questionnaires to put into a database, or double-enter the data). Some statistical packages include a warning message or tone for double data entry.
5. Run statistical frequencies on all items and scan the results for obvious anomalies. Are any data missing? Are there numbers that don’t seem right (for example, on a questionnaire when participants can only give answers in the range of 1-5, are there any numbers outside that group in your database?). Go back to your coding sheets and check all anomalous data.
6. Create codes for missing data. This allows you to locate errors quickly. For example, if participants refused to answer, or couldn’t answer a question, or only answered part of the study, you can build in codes to account for this. Remember to make each code distinct, so they cannot be confused with numbers elsewhere in database.
7. For questionnaires that have sections that should add up, ensure the answers tally.
8. When answers are missing or don’t add up, take time to locate incorrect answers. Check your database regularly as you add data, so that when errors arise it doesn’t take too much work to find them.
9. Make a note each time you clean your data, and flag where you’ve got up to each time you work on your database. That way, when you find errors, you will have a more precise idea where to locate the problem.
Trying a bit too hard
Gavin had completed a questionnaire study on use of hormone replacement therapy in women and their partners. He had used a number of standardised measures alongside a questionnaire he had created himself. Having finished his research, he was uncertain about how to report his findings, so he included analysis on every question within 14 tables. He also added 10 bar charts and two pie charts for good measure. Since he wasn’t sure what to say about them he let the data speak for itself. Unfortunately it didn’t, but Gavin was lucky enough to have a supervisor who explained he only needed a small number of tables and graphs to illustrate what he had found from his study.
w2. Nazareth I, Boynton P, King M. Problems with sexual function in people attending London general practitioners: cross sectional study. BMJ 2003;327:423-6.
w3. Tufte E. The visual display of quantitative information. Cheshire, CT: Graphics Press, 1983.
w4. Leudar I, Antaki C. Participant status in social psychological research. In: Ibanez T, Iniguez L, eds. Critical social psychology. London: Sage, 1997.
w5. Papagrigoriadis S.,.Heyman B. Patients’ views on follow up of colorectal cancer: implications for risk communication and decision making. Postgrad Med J 2003;79:403-7.
w6. Lau JT, Tsui HY, Wang QS. Effects of two telephone survey methods on the level of reported risk behaviours. Sex Transm Infect 2003;79:325-31.
w7. Evans L, Hughes-Webb P, Nicholas P, Fraser CL, Jamalapuram K, Hughes B. The use of e-mail by doctors in the West Midlands. J Telemed Telecare 2001;7:99-102.
w8. Rosenvinge JH, Laugerud S, Hjortdahl P. Trust in health websites: a survey among Norwegian internet users. J Telemed Telecare 2003;9:161-6.
w9. Boston NK, Boynton PM, Hood S. An inner city GP unit versus conventional care for elderly patients: prospective comparison of health functioning, use of services and patient satisfaction. Fam Pract 2001;18:141-8.
w10. Denzin M, Lincoln P. Handbook of qualitative research. London: Sage, 1994.
Method of delivery
Participants are sent a copy of the questionnaire by post and asked to complete it and return it to the researcher.w5
Quick and easy to distribute.
They are not useful for the study of very personal issues (without first giving the participant the option of taking part in the study) and have a notoriously low response rate since you are relying on the goodwill and co-operation of individuals.
Remember to enclose a detailed introductory letter, a complete, contact address, and a stamped addressed envelope so the participant does not have to pay postage.
You may need to send reminder letters and questionnaires to slow/non-responders.
The researcher calls participants and completes the questionnaire over the phone, with the researcher reading out the questions and recording the answers.w6
Quick and easy to complete.
Due to ethical constraints and sample bias these are used less within health research.
You cannot control for participant refusal, which is often high.
Not suitable for those with hearing problems.
Can become laborious if calling someone who is lonely and wants to talk.
Remember to contact participants by letter in advance of your call – and offer them a chance to opt-out of your study (and avoid your phone call).
Many ethics committees won’t permit a study where cold calling is the main design.
Questionnaires are sent to participants via email for completion.w7
Easy to design and send out.
Can keep track on who has responded and who hasn’t, and send reminders.
Only suitable for participants with email access, and who can download a questionnaire.
Can lead to confusion, where participants print out questionnaire and answer it by hand, rather than on the computer.
See telephone interview above. Participants need an introductory email announcing the research and an opt-out option. Follow data protection legislation, and check sending emails don't breach confidentiality.
By a website
The questionnaire is placed within a website and participants are directed to this and invited to complete it. w8
A simple questionnaire can be easily designed and placed within a website. Since sites offer more space, it’s possible to have more opportunities for qualitative feedback using this measure.
Participants are only those with access to the Internet. You may find they are a non-representative sample since they’ll have a special interest for visiting your site (e.g. your site is about testicular cancer and they have a particular view about it, or experience of illness). It is difficult to stop the same person answering the questionnaire a number of times over.
Check your site regularly to ensure you can access the questionnaire and that there aren’t any ‘bugs’ in it.
Encourage participants to report problems with accessing the questionnaire online.
Participant completion with researcher present
The researcher can answer questions the participants may have, but the participant answers the questions.w2
The researcher is on-hand to offer support and explain any questions participants might not understand.
They can also be sure that questionnaires are completed and collected.
Participants can inadvertently be ‘led’ by asking the researcher for advice on how to answer the questions.
Ensure your staff have training and support in how to deliver and code questionnaires and manage participants.16
The researcher asks the question and fills in the appropriate answers as directed by the participant.w9
The researcher can be certain the questionnaires are fully and accurately completed, and collected.
The researcher may ‘lead’ participants by their tone of voice or phrasing of questions.
Participants may not understand what is required of them and not answer in a ‘standardised way’.
If you are using standardised measures researchers have to read these out in exactly the same order as they appear written in the questionnaire.
Type of response required from participant on the questionnaire
You can analyse this data using…
Binary or yes/no answers
c2 (chi squared), Spearmans, Wilcoxon, Mann Whitney, Kruskal Wallis etc.
Rating or visual scales
Pearsons, t test, analysis of variance (ANOVA) etc.
Open-ended (free text) replies
Thematic content or discourse analysis.
- Correction Published: 05 August 2004; BMJ 329 doi:10.1136/bmj.329.7461.323-b
- Analysis And Comment Published: 03 August 2006; BMJ 333 doi:10.1136/bmj.333.7562.300
- Trump chooses Obamacare opponent to lead Health and Human ServicesBMJ November 30, 2016, 355 i6469; DOI: https://doi.org/10.1136/bmj.i6469
- Drug firms need to tackle affordabilityBMJ November 14, 2016, 355 i6096; DOI: https://doi.org/10.1136/bmj.i6096
- World Medical Association’s tainted president, Ketan DesaiBMJ November 10, 2016, 355 i5867; DOI: https://doi.org/10.1136/bmj.i5867
- Cancer drugs, survival, and ethicsBMJ November 09, 2016, 355 i5792; DOI: https://doi.org/10.1136/bmj.i5792
- WHO refers $29m Oxford research project to GMC for misconductBMJ November 07, 2016, 355 i5971; DOI: https://doi.org/10.1136/bmj.i5971
- Attitudes of multimorbid patients to surviving future acute illness and subsequent functional disability: A systematic review
- Survey research in anesthesiology: a field guide to interpretation
- Dementia among migrants and ethnic minorities in Italy: rationale and study protocol of the ImmiDem project
- Nationwide survey on volunteers' training in hospice and palliative care in Poland
- Development of a Tool for the Assessment of Nurses' Attitudes Toward Delirium
- Gaining an accurate reflection of the reality of palliative care through the use of free-text feedback in questionnaires: the AFTER study
- The role of a pharmacist in a hospice: a nationwide survey among hospice directors, pharmacists and physicians
- The use of internet-mediated cross-sectional studies in mental health research
- How to assess a survey report: a guide for readers and peer reviewers
- Rational use of medicines in the hospitals of Poland: role of the pharmacists
- Clinical dashboard: use in older adult mental health wards
- HIV testing: getting the message across--a survey of knowledge, attitudes and practice among non-HIV specialist physicians
- A Survey of Nurses' Beliefs About the Medical Emergency Team System in a Canadian Tertiary Hospital
- Clinical teaching in paediatrics: understanding perceptions, motives and concerns
- Medical students' attitudes towards abortion: a UK study
- Best Practices for Survey Research Reports: A Synopsis for Authors and Reviewers
- A guide for the design and conduct of self-administered surveys of clinicians
- Nurses' attitudes to a medical emergency team service in a teaching hospital
- Overcoming barriers to recruitment in health research
- Reaching beyond the white middle classes