How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews

BMJ 2002; 324 doi: http://dx.doi.org/10.1136/bmj.324.7337.573 (Published 9 March 2002)
Cite this as: BMJ 2002;324:573

Recent rapid responses

Rapid responses are electronic letters to the editor. They enable our users to debate issues raised in articles published on bmj.com. Although a selection of rapid responses will be included as edited readers' letters in the weekly print issue of the BMJ, their first appearance online means that they are published articles. If you need the url (web address) of an individual response, perhaps for citation purposes, simply click on the response headline and copy the url from the browser window.

Displaying 1-6 out of 6 published

31 January 2003

Dear Editor,

In response to Dr. Eysenbach's article, I feel that his research assessing consumer web searching methods and web page validation techniques, would have been further developed and substantiated if he had utilized a librarian, and the librarian's knowledge of consumer searching. Librarians are expert searchers, and provide a level of professionalism in assessing search methods and outcomes that his article does not represent. How can he state, "…search technique was often suboptimal…" or "Despite the somewhat ineffective search strategies used…" when no examples comparing "good" and "bad" search strategies are provided, or proof that he or his co-author have the credentials or experience to evaluate search strategies?

I commend Dr. Eysenbach on his awareness that research needs to be conducted assessing Internet users and the health information they gleam. I wonder what steps he plans to take in order to educate consumers in web searching and evaluating the validity of the information they find. A librarian has daily encounters with consumers seeking Internet information. So why not collaborate with a librarian, to assess user searching and instruct the users on the importance of knowing criteria to evaluate Web pages.

Librarians have conducted research for years on user Web searching techniques. When Dr. Eysenbach performed the background research to learn what has been studied, did he evaluate the library literature? I have included just a few of the many articles from librarian research that would be relevant to a study such as his. Your "theme issue" demonstrates an impressive awareness into a health concern where librarians can assist. In fact, a librarian would have provided an expert perspective in nearly all the research articles published in this issue.

Elizabeth M. LaRue
Head of Reference
Augustus C. Long Health Sciences Library, Columbia University, New York, NY 10032

1. Abate LE. Uncertainty in evaluating consumer health World Wide Web sites [Thesis MSLS]. Univ of NC at Chapel Hill, 1998.

2. Connor E. Using search engines to locate Web resources on women's health. Women's health on the Internet Haworth Press. United States, 2000:47-63 f.

3. Garrison S. Evaluating health Internet sites: a white paper's criteria. Medical Reference Services Quarterly;17(3):41-7.

4. Khalil FEM. Consumer health information: a brief critique on information needs and information seeking behaviour. Malaysian Journal of Library & Information Science;6(2):83-99.

5. Littleton DA. A review of strategies for finding health information on the World-Wide Web. Medical Reference Services Quarterly;17(2):51-5.

6. Lucas W, Topi H. Form and function: the impact of query term and operator usage on Web search results. relevancy of search engine results. Journal of the American Society for Information Science and Technology;53(2):95-108.

7. Nicholas D, Huntington P, Williams PE. Evaluating metrics for comparing the use of Web sites: a case study of two consumer health Web sites. NHS Direct Online and SurgeryDoor. Journal of Information Science;28(1):63-75.

8. Silbajoris C. Health information needs and the Internet : a survey of older adults [Thesis MSLS]. Univ of NC at Chapel Hill, 2000.

Competing interests:   None declared

Competing interests: None declared

Elizabeth M. La Rue, Head of Reference Augustus C. Long Health Sciences Library

Columbia University 10032

Click to like:

26 June 2002

Sir,

The findings in your study that Users of the internet explore only the first few links on general search engines when seeking health information is one with far reaching consequences.

It is a very well known fact that consumers in search of health information search on general purpose search engines rather than medical search engines or search engines of trustmarking/rating organisations .

This would imply that the first step in quality management of health information on the internet should start from search engines. Apart from employing technologies like the google pagerank[http://www.google.com/technology/index.html], which are prone for abuse, actual rating of websites should be done to ensure quality.

Your second finding that Consumers when assessing the credibility of a site primarily look for the source, a professional design, and a variety of other criteria. This clearly means that trustmarks or rating information form only a minor part of the credibility 'features' as far as the consumer is concerned.This denotes an absolute lack of consumer education in this sphere.

Thirdly you find that in practice, internet users do not check the "about us" sections of websites, try to find out who authors or owners of the site are, or read disclaimers or disclosure statements.This is also an impact of the lack of consumer education.

The need of the hour is to promote consumer education on e-health. Moreover search engines should be brought under a legally binding and acceptable general framework for selecting and presenting health related information on the internet.

References: [1]Vinod Scaria .Potential Problems with Trust-marking Health Information on the Net. e-journal of Cybermedicine and Medical Informatics 2002;1(1) URL: http://www.cybermed.150m.com/paper1.html

[2]Eysenbach G. Towards ethical guidelines for e-health: JMIR Theme Issue on eHealth Ethics. Journal of Medical Internet Research 2000;2(1):e7 URL: http://www.jmir.org/2000/1/e7/

Competing interests: None declared

Vinod Scaria, Center for Cybermedicine and Internet Research

calicut 673009

Click to like:

Editor--

I appreciate the response by Eysenbach [1] and apologize if I came across [2] as derogatory. I intended only to shift the emphasis from one of their findings (not actively searching for information on who stood behind sites) to another (cross checking information on different sites).

Eysenbach says:

[One] point of the paper is ... the difference between what people say they do and what they actually do. We noted that people told us ... that they check the source of the information ... but in the observational study we saw very few people actually looking at the source or even remembering where they had the information from.

we present our observations ... as a motivation to do more direct observational research rather than relying on survey data such as the Pew survey.

I agree wholeheartedly that in behavioral research, observations provide more accurate data than surveys. Eysenbach and Köhler should be commended on taking that approach [3], and further such research is needed.

However, Internet users do not always need to search actively to assess authoritativeness. BMJ articles, for example, clearly say "bmj.com" at the top and supply information about authors. Readers may not need to click their way to "BMA" or "Editorial board." And it may be enough for them just to determine that they consider a site trustworthy, especially if they have 8 or 9 questions to answer, without remembering what type of site it is.

Eysenbach also says:

we did not observe any participant cross-checking the answer he/she found on one site with facts presented on other sites: "In some cases, participants continued the search even after finding a page with an answer, generally because they did not understand the information encountered rather than because they did not trust the source".
The statement that he quotes implies that some participants did continue to search because they did not trust the source. If none did, then that is new and noteworthy information (and possibly another discrepancy between observation and self-report).

Robert C Hsiung, MD
Associate Professor of Clinical Psychiatry
University of Chicago
5737 S. University Ave.
Chicago, IL
60637-1507
USA
dr-bob@uchicago.edu

1. Eysenbach, G. Re: A more democratic approach to information [e-letter]. BMJ 2002 Jun 1. [Full text]

2. Hsiung, RC. A more democratic approach to information [e-letter]. BMJ 2002 May 30. [Full text]

3. Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002 Mar 9; 324 (7337): 573-7. [Full text]

Competing interests: None declared

Robert C Hsiung, Associate Professor of Clinical Psychiatry

University of Chicago, 5737 S. University Ave., Chicago, IL 60637-1507, USA

Click to like:

Robert Hsiung in his rapid response commented that "There may be a difference between what the investigators and the subjects think of as quality. The investigators, and researchers in general, may think in terms of scientific validity. Thus, they may consider the authority of the source to be paramount and may be dismayed that Internet users do not search for that information."

The point of the paper is what CONSUMERS think about quality and what THEIR yardsticks of quality is, and the difference between what people say they do and what they actually do. We noted that people told us in focus groups interviews that they check the source of the information and that the authority of the source is one of their yardsticks to decide whether or not to trust and apply health information, but in the observational study we saw very few people actually looking at the source or even remembering where they had the information from.

Another point of the paper is that consumers may use other yardsticks than researchers, such as design, as marker for trustworthiness and credibility. We report this as an observation without judging it (our remark about the "suboptimal" search strategy referred to the fact that few used Boolean operators or multiple seach terms) - and Hsiungs remark that researchers may be "dismayed" sounds slightly derogative. As researchers, we try to stay objective and report what we see. We are not "dismayed" - we just simply didn't see many people looking at the source statement of a website, and report this as a finding worth to be explored in more representative studies, and as a finding which may have implications for educational interventions and also health information providers on the web. Another derogative remark in Hsiungs reply is that "Eysenbach and Koehler note that their subjects 'could verify and cross check the information on different sites,' but underemphasize this more democratic approach to information".

I don't see what leads Hsiung to believe that we "underestimate" the fact that the beauty of the web lies in the fact that consumers can get information from multiple sources - we don't, that's why we mentioned it. We simply report that we didn't observe any of our participants actually doing it, although focus group participants told us before that the comparison of facts on different sites is something that they like about the web. In the usability experiment we did not observe any participant cross-checking the answer he/she found on one site with facts presented on other sites: "In some cases, participants continued the search even after finding a page with an answer, generally because they did not understand the information encountered rather than because they did not trust the source". This may well be due to the several limitations of our method, discussed in the paper, and needs to be verified by other research. Still, we believe that it is worth exploring how consumers actually search the web, and to what degree they cross-check and aggregate information from different websites. For the same reason, we argued in our recently published systematic review about empiral studies assessing the quality of health information (1) that the quality criterion of "completeness" of information on a website (introduced by Impicciatore and recooked in many poor "me-too" studies) is a quastionable criterion (in other words: it is nonsense).

In our BMJ paper we present our observations, and as a motivation to do more direct observational research rather than relying on survey data such as the Pew survey. It is a very significant difference to ask people a closed question such as "If you find similar information on different sites, do the similarities give you more confidence in the sites (yes/no)" (it doesn't surprise me at all that the majority answers YES here!) and actually observing people HOW they retrieve information and under which circumstances they cross-check information. This is where qualitative research is needed - quantitative statements such as the one cited by Hsiung from the Pew survey do not help us much to understand how people use health information and how we can maximize the positive effects of health communication on the web.

Reference

1. Gunther Eysenbach, John Powell, Oliver Kuss, Eunryoung Sa. Empirical studies assessing the quality of health information for consumers on the World Wide Web: A systematic review. JAMA, 2002; 287: 2691-2700. http://jama.ama-assn.org/issues/v287n20/abs/jrv10005.html

Competing interests: None declared

Gunther Eysenbach, Associate Professor, Centre for Global eHealth Innovation

Toronto

Click to like:

Editor--

Eysenbach and Köhler [1] report the reassuring finding that their subjects were "very successful in retrieving information," yet the investigators refer to the search techniques used by their subjects as "suboptimal."

There may be a difference between what the investigators and the subjects think of as "quality." The investigators, and researchers in general, may think in terms of scientific validity. Thus, they may consider the "authority" of the source to be paramount and may be dismayed that Internet users do not search for that information. On the other hand, the subjects, and Internet users in general, may think in terms of trustworthiness in a broader sense. The authority of the source contributes to the trustworthiness of the information provided, but so do other criteria reported by Eysenbach and Köhler, such as accountability of the site owner (represented by his or her picture and e-mail address), being linked to by other sites, and absence of advertising. Researchers may want the definitive answer to a question, but typical Internet users may be satisfied with "an answer that they [feel] confident about."

Eysenbach and Köhler note that their subjects "could verify and cross check the information on different sites," but underemphasize this more democratic approach to information. The Pew Internet & American Life Project recently reported [2] that "a majority of multiple-site searchers who found similar information on different sites say the similarities gave them more confidence in the sites." And discussion on Psycho-Babble [3, 4], the peer support group that I host, has articulated this even with respect to single sites:

The search for truth reminds me of Hegel: it is neither the "thesis" (the claim by the manufacturer that the medication is some sort of wonder drug) nor the "antithesis" (the claim by someone who blames all their problems on the medication), but rather a "synthesis" (a sober analysis of both positive and negative aspects). Information which is balanced and fair is trustworthy, whereas that which comes from either advocacy viewpoint is suspect.
Eysenbach and Köhler's subjects may not have searched for information on who stood behind the sites they searched -- but they did not need to! They successfully answered questions in an average of 5 minutes 42 seconds, and the proof of the pudding is in the eating.

Robert C Hsiung, MD
Associate Professor of Clinical Psychiatry
University of Chicago
5737 S. University Ave.
Chicago, IL
60637-1507
USA
dr-bob@uchicago.edu

1. Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002 Mar 9; 324 (7337): 573-7. [Full text]

2. Fox S, Rainie L. Vital decisions. Pew Internet & American Life Project. 2002 May 22. [Full text]

3. Hsiung RC. The best of both worlds: An online self-help group hosted by a mental health professional. CyberPsychology & Behavior 2000; 3 (6): 935-950. [Full text]

4. Hsiung, RC. Ownership: Rights and responsibilities [e-letter]. BMJ 2001 Nov 12. [Full text]

Competing interests: None declared

Robert C Hsiung, Associate Professor of Clinical Psychiatry

University of Chicago, 5737 S. University Ave., Chicago, IL, 60637-1507, USA

Click to like:

Focus on quality of website information neglects issue of internet usage

Editor- We read with interest the recent internet ‘theme issue’ (9 March) which concentrated on the quality and accuracy of web-based health information. However, this focus begs the question: are patients actually using the internet in the first place?

We recently conducted an audit of information sources used by people with diabetes attending hospital clinics. A single interviewer (JLR) administered a short questionnaire to 70 consecutive attendees of two diabetes centres in Glasgow (Gartnavel General Hospital and Glasgow Royal Infirmary). Five others were approached but declined to take part. Of those who used the internet [13/ 70 (19%)] only one used the home page for Diabetes UK while the others stated that they relied on results produced by general search engines. Only three stated that the internet was their favourite way for accessing diabetes information. Age appeared to be a major determinant of internet usage: younger patients (<_40 years="years" were="were" more="more" likely="likely" to="to" use="use" the="the" internet="internet" than="than" older="older" patients="patients" _="_"/>40 years) (Chi squared =7.62 p<_0.01. furthermore="furthermore" in="in" a="a" _2="_2" x="x" _3="_3" table="table" comparing="comparing" those="those" with="with" no="no" qualifications="qualifications" school-level="school-level" or="or" higher="higher" educational="educational" who="who" had="had" achieved="achieved" levels="levels" of="of" were="were" more="more" likely="likely" to="to" use="use" the="the" internet="internet" chi="chi" squared="6.61" p0.037.="p0.037." patients="patients" aged="aged" _60="_60" over="over" _56="_56" only="only" one="one" patient="patient" ever="ever" used="used" internet.="internet." age="age" distribution="distribution" respondents="respondents" our="our" sample="sample" was="was" generally="generally" accord="accord" prevalence="prevalence" reported="reported" previous="previous" population="population" screening="screening" studies.1="studies.1" p="p"/> These results suggest that few patients in our clinics use the internet and those who do are not necessarily gaining access to the best and most accurate information. Guidance from health professionals may therefore be required: perhaps a printed list of ‘recommended’ sites should be available at diabetes clinics? Given current levels of internet access, time and resources invested in developing educational websites for people with diabetes may only fulfill the needs of a small number of young and relatively well educated individuals.

1 Neil HAW, Gatling W, Mather HM, Thompson AV, Thorogood M, Fowler GH,Hill RD, Mann JI. The Oxford Community Diabetes Study: evidence for an increase in the prevalence of known diabetes in Great Britain. Diabetic Medicine 1987;4:539-543.

Julia L Robertson (medical student), 1, 2
Florence Brown (diabetes nurse specialist), 1
Greg Jones (consultant diabetologist), 1
Colin G Perry (lecturer)1, 2
John R Petrie (senior lecturer), 1, 2

1Diabetes Centres
North Glasgow University Hospitals NHS Trust and

2Department of Medicine
University of Glasgow, Queen Elizabeth Building, Glasgow Royal Infirmary, Glasgow G31 2ER
email jrp1s@clinmed.gla.ac.uk

note:
a)'older patients (>40)' should read 'greater or equal to' in symbol form but this symbol was unable to be inserted into this email.
b) when 'Chi squared' appears twice this should be in symbol form but again, this symbol was unable to be inserted into this email.

Competing interests: None declared

John R Petrie, Senior Lecturer

Julia L. Robertson, Florence Brown, Greg Jones, and Colin G. Perry

Queen Elizabeth Building, Glasgow Royal Infirmary, Glasgow G31 2ER

Click to like:

THIS WEEK'S POLL