How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviewsBMJ 2002; 324 doi: http://dx.doi.org/10.1136/bmj.324.7337.573 (Published 09 March 2002) Cite this as: BMJ 2002;324:573
- Gunther Eysenbach (), senior researcher,
- Christian Köhler, researcher
- Unit for Cybermedicine and eHealth, Department of Clinical Social Medicine, University of Heidelberg, Bergheimer Str 58, 69115 Heidelberg, Germany
- Correspondence to: G Eysenbach
- Accepted 30 January 2002
Objectives: To describe techniques for retrieval and appraisal used by consumers when they search for health information on the internet.
Design: Qualitative study using focus groups, naturalistic observation of consumers searching the world wide web in a usability laboratory, and in-depth interviews.
Participants: A total of 21 users of the internet participated in three focus group sessions. 17 participants were given a series of health questions and observed in a usability laboratory setting while retrieving health information from the web; this was followed by in-depth interviews.
Setting: Heidelberg, Germany.
Results: Although their search technique was often suboptimal, internet users successfully found health information to answer questions in an average of 5 minutes 42 seconds (median 4 minutes 18 seconds) per question. Participants in focus groups said that when assessing the credibility of a website they primarily looked for the source, a professional design, a scientific or official touch, language, and ease of use. However, in the observational study, no participants checked any “about us” sections of websites, disclaimers, or disclosure statements. In the post-search interviews, it emerged that very few participants had noticed and remembered which websites they had retrieved information from.
Conclusions: Further observational studies are needed to design and evaluate educational and technological innovations for guiding consumers to high quality health information on the web.
What is already known on this topic
What is already known on this topic Little is known about how consumers retrieve and assess the quality of health information on the internet
Qualitative data are needed to design educational and technological innovations to guide consumers to high quality health information
What this study adds
What this study adds Users of the internet explore only the first few links on general search engines when seeking health information
Consumers say that when assessing the credibility of a site they primarily look for the source, a professional design, and a variety of other criteria
In practice, internet users do not check the “about us” sections of websites, try to find out who authors or owners of the site are, or read disclaimers or disclosure statements
Very few internet users later remember from which websites they retrieved information or who stood behind the sites
Little is known about how consumers retrieve and assess health information on the world wide web. Some surveys have elicited data by using semistructured questionnaires or focus groups,1–3 but little (if any) unobtrusive observational research has been done to explore how consumers are actually surfing the web. Although several criteria for quality of health websites have been proposed—including disclosure of site owners, authors, and update cycle 4 5 —little or nothing is known about whether and to what degree such markers are recognised or even looked at by consumers or what other credibility markers consumers are looking for. We aimed to obtain qualitative and semiquantitative data to generate some hypotheses on how consumers might search for and appraise health information.
We used multiple methods of data collection that are commonly used in studies of human-computer interactions,6 combining focus groups,7 naturalistic observation of consumers searching the internet, and post-search in-depth interviews. Two researchers independently analysed transcripts by using N5 (NUD*IST 5.0; QSR International, Melbourne) with the grounded theory approach.8
Participants in the focus groups and the observational study were mostly healthy volunteers recruited through newspaper advertisements seeking people who had already searched for health information on the web. We selected them on a first come first served basis.
We held three focus group sessions with 6-8 participants each (21 participants in total: five men, 16 women; mean and median age 37, range 19-71 years) in March 2001. Self reported internet experience of the participants ranged from 17 to 84 months (mean 46 months; median 42 months). Each session was facilitated by a skilled moderator and lasted about two to three hours. The aim of the focus groups was to identify the needs, expectations, and problems of consumers with respect to health information on the internet, with emphasis on how consumers appraise the quality of such information. We excluded health professionals from the focus groups. We audiotaped, videotaped, and transcribed the focus group sessions.
Naturalistic observation of search and retrieval processes
We conducted the observational study in September 2001, with the main goal of establishing how consumers search the web, how successful they are, and what markers for quality they use. We gave eight or nine health related questions individually to six participants from the focus groups and 11 further participants and observed them in a usability laboratory setting while they searched the internet for answers. Six of the participants were men and 11 were women, including three nurses (mean age 38, median age 36, range 18-71 years). Their internet experience ranged from two months to 12 years (mean 33 months, median 24 months). We chose questions from the Heidelberg Database of Patient Questions (HEIDPAQ), an anonymised repository of health related questions asked by consumers on the internet to an “ask the doctor” service. Example questions were “If you want to travel to Australia, do you need malaria prophylaxis?” and “What is the definition of being overweight?” We gave participants a maximum of 20 minutes per question to find an answer that they felt confident about. We instructed participants to think aloud during the searches and to use the same techniques that they would use at home. No specific search engine or method was prescribed or recommended, and participants started with an empty browser screen.
The usability laboratory consists of a computer with special software (Surf Spy; Bysoft Data AB, Byske, Sweden) to log web addresses visited, a digital video camera to record the participant and the computer screen, and one or two observers sitting in the background and taking field notes.
We conducted semistructured interviews with all 17 participants in the observational study immediately after their searches to elicit decision making processes and criteria for selection of a particular site.
Transcripts of focus groups and interviews
We analysed the transcripts of the focus groups and interviews for emerging themes concerning markers used by consumers to evaluate the quality of health information (box 1) and general statements about the use of the internet to obtain health information (box 2).
Box 1: Quotes from focus groups and in-depth interviews—criteria for credibility of healthcare information on the internet
Authority of source
“I want to know where that information comes from. Sometimes it is hard to detect who is responsible for the content, and this is bugging me”
“I consider it to be reliable if information is from public institutions or scientific publications”
“I certainly trust more an official website of an organisation or association rather than a private site”
“I would like to know whether this is the opinion of a single person or an institution, whether the content is selected according to scientific criteria or whether it is based on a personal experience”
Layout and appearance
“The presentation of the site is very important, so that it appears professional”
“The design and the advertising, the whole appearance must be pleasing. That's very important”
“I have been on a [AIDS] site that was not too bad actually, but there have been some strange banners on top [advertising banners for pornographic websites]. This, of course, badly affected the credibility of that site”
“To me, the text shouldn't use too much professional terminology. It should appear in a language a medical layperson can understand without great difficulties”
“The text should be in a respectable and comprehensible diction, not too sensational.”
“If a trustworthy site provides links to other sites, I assume they are trustworthy as well”
Picture of the site owner
“Perhaps a picture of the owner of the site would be helpful. It would be a plus if the site owner would have a pleasing face or appears likeable”
“I think it is the first impression. If there is a picture of a honestly smiling man, that would be very important”
“One should have the ability to contact the provider, such as by email. And the owner should respond”
Credentials and qualifications
“If possible, the site owner should provide evidence of his credentials, that he is qualified to make such statements. For example, references to his previous work, or his curriculum vitae”
Updating of content
“I have seen pages that say ‘Last updated in 1998.’ In this case, I don't need to go any further. But I have seen on the site of that institute of tropical medicine that they were giving advice concerning anthrax. In this case I know that this site is updated frequently”
[The interview was conducted shortly after the first cases of mail contaminated with anthrax in the United States occurred]
Quality seal and third party endorsements
“The content should be checked by the Federal Department of Health or any authority that can say ‘Yes, this is OK.’”
Box 2: Quotes from focus groups and in-depth interviews—use of the internet for obtaining health information
“In particular, for information retrieval the internet is really, really useful. I can easily go for a second, third, or probably fifteenth opinion about an issue and see what comes up”
“I can get three types of health information: I can print out information from the internet, I can read some literature, and I can talk to my doctor. Then I can see whether this information is congruent or differs. So, as a layperson, I obtain some kind of medical know-how and I can disagree with my doctor”
“The internet encourages you just to skim the text. You don't get as deep into it as you would do with printed material”
“Remember how, what for, and by whom the internet has been developed. It has been developed by 18 year old chaps. We shouldn't forget that these young fellows are putting up this crap without having a clue about what knowledge really means”RETURN TO TEXT
Several criteria to assess credibility emerged from the interviews. Websites from official authorities, a professional layout, understandable and professional writing, and citation of scientific references were the most often mentioned criteria for trustworthiness. Some participants also mentioned accessibility and certain site features as quality criteria—in particular, many wished to have “a quick overview over what is on the site” (site map); search capabilities; a user friendly, uncluttered, speedy interface; the ability to send an email to the site owner; and outbound links to further recommended websites. Some participants expressed a desire to see a picture of the site owner or author—if the owner looked trustworthy, the site content was perceived to be credible as well. Other markers for quality mentioned were that the text should “sound plausible” or “sound scientific” and that treatment discussed “doesn't have any side effects” or is “natural, no chemistry.” Some participants mentioned that some form of controlling authority or an endorsement by a third party would be helpful.
A few participants noted that it is easier to assess the quality of information on the internet than elsewhere because they could verify and cross check the information on different sites. In general, consumers said that they enjoyed the new opportunities that the internet is opening up to verify what the physician says and to look for alternatives, but at the same time they stressed that they would always verify what they found on the web with their physician.
Observations during the retrieval experiment
We could do a detailed analysis of the search process for 16 of the 17 participants; one participant's log file was accidentally deleted. All 16 participants together visited 763 different web pages from 375 different sites if multiple visits of a given participant to the same page or site were counted only once, or a total of 646 unique web pages from 271 unique sites if all pages and sites were counted only once even if they were visited by more than one participant. In addition, participants used eight different search engines (Google, AltaVista, Yahoo, Web.de, dr-antonius.de, Abacho, Lycos, and Fireball.de) for a total of 280 search queries. On average, participants remained on each website for only 1 minute and 9 seconds (median 37 seconds); the three nurses spent slightly less time per page (mean 40 seconds, median 28 seconds). Although some of the websites carried a Health on the Net logo, none of the participants clicked on this logo.
Search strategy and effectiveness of searches
None of the participants used medical portals or the sites of medical societies or libraries as a starting point. Rather, they used search engines in an attempt to find relevant pages, except in two cases where for a given question participants tried to guess a web address—for example, with the pattern http://www.disease.de/.
In most cases we felt that the search strategy used by participants was somewhat suboptimal. For all answers it would have been most effective to enter combinations of words (for example, “Australia malaria prophylaxis”), but only 99 (35%) of the 280 search queries consisted of more than one search term. Only one participant used explicit Boolean operators (in this case “AND”). Phrase searches (words enclosed in quotes to search for documents where they appear together) were used by only five participants in 10 (3.5%) of 280 searches.
As shown in the figure, participants usually chose one of the first results displayed by the search engine and then rephrased their search rather than turning to the second page and exploring further results. In 281 (97.2%) of 289 clicks on a link, the link ranked among the first 10 search results; in 206 (71.3%) of 289 cases the participants followed a link ranked among the first five results (figure). Only nine participants ever looked at the second or following pages of the search result pages, and only five of them ever clicked a link on the second or following pages.
One elderly participant thought that the 10 search results displayed on the first page were all that existed. Further suboptimal uses of the search engine included one participant who desperately tried to find German sites when entering “HIV test” but found only English pages, without realising that the search engine had an option to limit results to pages written in a particular language. Instead, he used the translation tool of Google to translate the resulting pages. None of the 13 German participants who self reported good command of English entered search terms in English to deliberately find non-German sources.
Despite the somewhat ineffective search strategies used, participants were very successful in retrieving information that enabled them to answer the questions. The 17 participants could find no web page that contained the answer to the question for only seven of the total of 136 questions given to them. (Five of these seven concerned a question about whether a 16 year old girl needs parental agreement for an abortion.) On average, it took only 5 minutes 42 seconds (median 4 minutes 18 seconds; range 38 seconds to 20 minutes) per question to find an answer. Nurses had very similar search times to non-nurses (mean 5 minutes 37 seconds v 5 minutes 45 seconds; median 4 minutes 48 seconds v 4 minutes 16 seconds). We found no correlation between internet experience and search time. In some cases, participants continued the search even after finding a page with an answer, generally because they did not understand the information encountered rather than because they did not trust the source. It was not within the scope of this study to evaluate the quality of answers retrieved by consumers, but in general we did not encounter any gross misinformation—wrong answers given by consumers were more likely to have occurred through misunderstanding on the part of the consumer.
Did consumers pay attention to the sources?
In most cases, participants used websites that they said they had not known before. Contrary to the statements made in the focus groups, in practice we observed that none of the participants actively searched for information on who stood behind the sites or how the information had been compiled; often they did not even visit the home page. In the course of the post-search interview we asked 12 of the participants, while going step by step through their answers, from which source they thought they had retrieved the particular information. Participants could correctly reproduce the name of the website or the company or organisation who stood behind the site for only 20.9% (18/86) of the answers; they could tell to which broad category the site owner belonged to (for example, government agency, public institution, university, commercial organisation, private person, self help group) for only 23.2% (20/86). In all cases, the reason for this failure was not that the site did not disclose such information but that users did not pay attention to the origin of the information.
In six cases, participants ended up on an Austrian or Swiss website when searching for legal information about abortion. Only half of them realised this fact, although these sites contained information that was not necessarily applicable in Germany.
The results of this study indicate that consumers search for and appraise information in a different way than is implicitly assumed in many studies in which investigators assessed the quality of information on the web by entering a single search phrase and systematically evaluating the quality of all hits. This study is, to our knowledge, the first observational study carried out in a usability laboratory to investigate the retrieval strategies of people searching for health information on the web. Participants were very successful in finding answers to health questions by trying various search terms, exploring the first few hits by cursory examination of the content of the page, and iteratively refining their search strategy. However, few participants took notice and later remembered from which websites they retrieved information or who stood behind the sites.
The study has several important limitations. Firstly, we observed only a small number of participants in a somewhat artificial environment under experimental conditions. We cannot rule out the possibility that participants may have felt time pressure and thus put less emphasis on checking the quality and source. We tried to minimise this by explicitly instructing them to take their time and try to retrieve the correct answer rather than a quick answer. Secondly, as answering the questions did not have any direct impact on the participants, people in a real setting with a greater stake in the outcome of the search might care more about quality and therefore more actively look for markers of quality. Thirdly, we do not know to what degree the sample we tested was representative, but it is likely that we had a relatively experienced sample as people volunteered to participate in this study, we had three nurses in the sample, and some of our participants had taken part in the preceding focus groups. Although the nurses in our sample did not perform notably better than other participants, future studies with more participants and greater statistical power could explore whether health professionals use different strategies for search and appraisal than do consumers, are more successful than consumers, and if so why.
More observational studies are needed to design and evaluate educational and technological innovations for guiding consumers to high quality health information on the web. This is one of the important challenges in the age of “consumer health informatics.”9
The Heidelberg Database of Patient Questions (HEIDPAQ) largely contains “ask the doctor” questions kindly supplied by Netdoktor.de and Netdoctor.co.uk
Contributors: GE had the idea for the study, obtained funding, planned and moderated the focus group sessions, directed the design and analysis of the usability study, participated in most usability tests and in-depth interviews, wrote the first draft of the paper, and is the guarantor. CK participated in the design of the usability study, developed the study software, conducted all usability tests and in-depth interviews, and analysed the data. Both authors performed a qualitative analysis of the transcripts.
Funding Partly supported by the EU Safer Internet Action Plan (http://europa.eu.int/iap/), conducted as part of the MedCERTAIN project (http://www.medcertain.org/), a joint project of the Department of Clinical Social Medicine, University of Heidelberg, Germany; the Institute for Learning and Research Technology (ILRT), University of Bristol, UK; and the Finnish Office for Health Care Technology Assessment (FinOHTA) at the Finnish National Research and Development Centre for Welfare and Health (STAKES).
Competing interests None declared.