How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews
BMJ 2002; 324 doi: https://doi.org/10.1136/bmj.324.7337.573 (Published 09 March 2002) Cite this as: BMJ 2002;324:573All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Sir,
The findings in your study that Users of the internet explore only
the first few links on general search engines when seeking health
information is one with far reaching consequences.
It is a very well known fact that consumers in search of health
information search on general purpose search engines rather than medical
search engines or search engines of trustmarking/rating organisations .
This would imply that the first step in quality management of health
information on the internet should start from search engines. Apart from
employing technologies like the google
pagerank[http://www.google.com/technology/index.html], which are prone for
abuse, actual rating of websites should be done to ensure quality.
Your second finding that Consumers when assessing the credibility of
a site primarily look for the source, a professional design, and a
variety of other criteria. This clearly means that trustmarks or rating
information form only a minor part of the credibility 'features' as far as
the consumer is concerned.This denotes an absolute lack of consumer
education in this sphere.
Thirdly you find that in practice, internet users do not check the
"about us" sections of websites, try to find out who authors or owners of
the site are, or read disclaimers or disclosure statements.This is also an
impact of the lack of consumer education.
The need of the hour is to promote consumer education on e-health.
Moreover search engines should be brought under a legally binding and
acceptable general framework for selecting and presenting health related
information on the internet.
References:
[1]Vinod Scaria .Potential Problems with Trust-marking Health Information
on the Net.
e-journal of Cybermedicine and Medical Informatics 2002;1(1) URL:
http://www.cybermed.150m.com/paper1.html
[2]Eysenbach G. Towards ethical guidelines for e-health: JMIR Theme
Issue on eHealth Ethics.
Journal of Medical Internet Research 2000;2(1):e7
URL: http://www.jmir.org/2000/1/e7/
Competing interests: No competing interests
Editor--
I appreciate the response by Eysenbach [1] and apologize if I came across [2] as derogatory. I intended only to shift the emphasis from one of their findings (not actively searching for information on who stood behind sites) to another (cross checking information on different sites).
Eysenbach says:
[One] point of the paper is ... the difference between what people say they do and what they actually do. We noted that people told us ... that they check the source of the information ... but in the observational study we saw very few people actually looking at the source or even remembering where they had the information from.
we present our observations ... as a motivation to do more direct observational research rather than relying on survey data such as the Pew survey.
I agree wholeheartedly that in behavioral research, observations provide more accurate data than surveys. Eysenbach and Köhler should be commended on taking that approach [3], and further such research is needed.
However, Internet users do not always need to search actively to assess authoritativeness. BMJ articles, for example, clearly say "bmj.com" at the top and supply information about authors. Readers may not need to click their way to "BMA" or "Editorial board." And it may be enough for them just to determine that they consider a site trustworthy, especially if they have 8 or 9 questions to answer, without remembering what type of site it is.
Eysenbach also says:
we did not observe any participant cross-checking the answer he/she found on one site with facts presented on other sites: "In some cases, participants continued the search even after finding a page with an answer, generally because they did not understand the information encountered rather than because they did not trust the source".
The statement that he quotes implies that some participants did continue to search because they did not trust the source. If none did, then that is new and noteworthy information (and possibly another discrepancy between observation and self-report).
Robert C Hsiung, MD
Associate Professor of Clinical Psychiatry
University of Chicago
5737 S. University Ave.
Chicago, IL
60637-1507
USA
dr-bob@uchicago.edu
1. Eysenbach, G. Re: A more democratic approach to information [e-letter]. BMJ 2002 Jun 1. [Full text]
2. Hsiung, RC. A more democratic approach to information [e-letter]. BMJ 2002 May 30. [Full text]
3. Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002 Mar 9; 324 (7337): 573-7. [Full text]
Competing interests: No competing interests
Robert Hsiung in his rapid response commented that "There may be a
difference between what the investigators and the subjects think of as
quality. The investigators, and researchers in general, may think in terms
of scientific validity. Thus, they may consider the authority of the
source to be paramount and may be dismayed that Internet users do not
search for that information."
The point of the paper is what CONSUMERS think about quality and what
THEIR yardsticks of quality is, and the difference between what people say
they do and what they actually do. We noted that people told us in focus
groups interviews that they check the source of the information and that
the authority of the source is one of their yardsticks to decide whether
or not to trust and apply health information, but in the observational
study we saw very few people actually looking at the source or even
remembering where they had the information from.
Another point of the paper is that consumers may use other yardsticks than
researchers, such as design, as marker for trustworthiness and
credibility. We report this as an observation without judging it (our
remark about the "suboptimal" search strategy referred to the fact that
few used Boolean operators or multiple seach terms) - and Hsiungs remark
that researchers may be "dismayed" sounds slightly derogative. As
researchers, we try to stay objective and report what we see. We are not
"dismayed" - we just simply didn't see many people looking at the source
statement of a website, and report this as a finding worth to be explored
in more representative studies, and as a finding which may have
implications for educational interventions and also health information
providers on the web.
Another derogative remark in Hsiungs reply is that "Eysenbach and Koehler
note that their subjects 'could verify and cross check the information on
different sites,' but underemphasize this more democratic approach to
information".
I don't see what leads Hsiung to believe that we "underestimate" the fact
that the beauty of the web lies in the fact that consumers can get
information from multiple sources - we don't, that's why we mentioned it.
We simply report that we didn't observe any of our participants actually
doing it, although focus group participants told us before that the
comparison of facts on different sites is something that they like about
the web. In the usability experiment we did not observe any participant
cross-checking the answer he/she found on one site with facts presented on
other sites: "In some cases, participants continued the search even after
finding a page with an answer, generally because they did not understand
the information encountered rather than because they did not trust the
source". This may well be due to the several limitations of our method,
discussed in the paper, and needs to be verified by other research. Still,
we believe that it is worth exploring how consumers actually search the
web, and to what degree they cross-check and aggregate information from
different websites. For the same reason, we argued in our recently
published systematic review about empiral studies assessing the quality of
health information (1) that the quality criterion of "completeness" of
information on a website (introduced by Impicciatore and recooked in many
poor "me-too" studies) is a quastionable criterion (in other words: it is
nonsense).
In our BMJ paper we present our observations, and as a motivation to do
more direct observational research rather than relying on survey data such
as the Pew survey. It is a very significant difference to ask people a
closed question such as "If you find similar information on different
sites, do the similarities give you more confidence in the sites (yes/no)"
(it doesn't surprise me at all that the majority answers YES here!) and
actually observing people HOW they retrieve information and under which
circumstances they cross-check information. This is where qualitative
research is needed - quantitative statements such as the one cited by
Hsiung from the Pew survey do not help us much to understand how people
use health information and how we can maximize the positive effects of
health communication on the web.
Reference
1. Gunther Eysenbach, John Powell, Oliver Kuss, Eunryoung Sa.
Empirical studies assessing the quality of health information for
consumers on the World Wide Web: A systematic review. JAMA, 2002; 287:
2691-2700. http://jama.ama-assn.org/issues/v287n20/abs/jrv10005.html
Competing interests: No competing interests
Editor--
Eysenbach and Köhler [1] report the reassuring finding that their subjects were "very successful in retrieving information," yet the investigators refer to the search techniques used by their subjects as "suboptimal."
There may be a difference between what the investigators and the subjects think of as "quality." The investigators, and researchers in general, may think in terms of scientific validity. Thus, they may consider the "authority" of the source to be paramount and may be dismayed that Internet users do not search for that information. On the other hand, the subjects, and Internet users in general, may think in terms of trustworthiness in a broader sense. The authority of the source contributes to the trustworthiness of the information provided, but so do other criteria reported by Eysenbach and Köhler, such as accountability of the site owner (represented by his or her picture and e-mail address), being linked to by other sites, and absence of advertising. Researchers may want the definitive answer to a question, but typical Internet users may be satisfied with "an answer that they [feel] confident about."
Eysenbach and Köhler note that their subjects "could verify and cross check the information on different sites," but underemphasize this more democratic approach to information. The Pew Internet & American Life Project recently reported [2] that "a majority of multiple-site searchers who found similar information on different sites say the similarities gave them more confidence in the sites." And discussion on Psycho-Babble [3, 4], the peer support group that I host, has articulated this even with respect to single sites:
The search for truth reminds me of Hegel: it is neither the "thesis" (the claim by the manufacturer that the medication is some sort of wonder drug) nor the "antithesis" (the claim by someone who blames all their problems on the medication), but rather a "synthesis" (a sober analysis of both positive and negative aspects). Information which is balanced and fair is trustworthy, whereas that which comes from either advocacy viewpoint is suspect.
Eysenbach and Köhler's subjects may not have searched for information on who stood behind the sites they searched -- but they did not need to! They successfully answered questions in an average of 5 minutes 42 seconds, and the proof of the pudding is in the eating.
Robert C Hsiung, MD
Associate Professor of Clinical Psychiatry
University of Chicago
5737 S. University Ave.
Chicago, IL
60637-1507
USA
dr-bob@uchicago.edu
1. Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002 Mar 9; 324 (7337): 573-7. [Full text]
2. Fox S, Rainie L. Vital decisions. Pew Internet & American Life Project. 2002 May 22. [Full text]
3. Hsiung RC. The best of both worlds: An online self-help group hosted by a mental health professional. CyberPsychology & Behavior 2000; 3 (6): 935-950. [Full text]
4. Hsiung, RC. Ownership: Rights and responsibilities [e-letter]. BMJ 2001 Nov 12. [Full text]
Competing interests: No competing interests
Focus on quality of website information neglects issue of internet
usage
Editor- We read with interest the recent internet ‘theme issue’ (9
March) which concentrated on the quality and accuracy of web-based health
information. However, this focus begs the question: are patients actually
using the internet in the first place?
We recently conducted an audit of information sources used by people
with diabetes attending hospital clinics. A single interviewer (JLR)
administered a short questionnaire to 70 consecutive attendees of two
diabetes centres in Glasgow (Gartnavel General Hospital and Glasgow Royal
Infirmary). Five others were approached but declined to take part. Of
those who used the internet [13/ 70 (19%)] only one used the home page for
Diabetes UK while the others stated that they relied on results produced
by general search engines. Only three stated that the internet was their
favourite way for accessing diabetes information. Age appeared to be a
major determinant of internet usage: younger patients (<_40 years="years" were="were" more="more" likely="likely" to="to" use="use" the="the" internet="internet" than="than" older="older" patients="patients" _="_"/>40 years) (Chi
squared =7.62 p<_0.01. furthermore="furthermore" in="in" a="a" _2="_2" x="x" _3="_3" table="table" comparing="comparing" those="those" with="with" no="no" qualifications="qualifications" school-level="school-level" or="or" higher="higher" educational="educational" who="who" had="had" achieved="achieved" levels="levels" of="of" were="were" more="more" likely="likely" to="to" use="use" the="the" internet="internet" chi="chi" squared="6.61" p0.037.="p0.037." patients="patients" aged="aged" _60="_60" over="over" _56="_56" only="only" one="one" patient="patient" ever="ever" used="used" internet.="internet." age="age" distribution="distribution" respondents="respondents" our="our" sample="sample" was="was" generally="generally" accord="accord" prevalence="prevalence" reported="reported" previous="previous" population="population" screening="screening" studies.1="studies.1" p="p"/> These results suggest that few patients in our clinics use the
internet and those who do are not necessarily gaining access to the best
and most accurate information. Guidance from health professionals may
therefore be required: perhaps a printed list of ‘recommended’ sites
should be available at diabetes clinics? Given current levels of internet
access, time and resources invested in developing educational websites for
people with diabetes may only fulfill the needs of a small number of young
and relatively well educated individuals.
1 Neil HAW, Gatling W, Mather HM, Thompson AV, Thorogood M, Fowler GH,Hill RD, Mann JI. The Oxford Community
Diabetes Study: evidence for an increase in the
prevalence of known diabetes in Great Britain. Diabetic
Medicine 1987;4:539-543.
Julia L Robertson (medical student), 1, 2
Florence Brown (diabetes nurse specialist), 1
Greg Jones (consultant diabetologist), 1
Colin G Perry (lecturer)1, 2
John R Petrie (senior lecturer), 1, 2
1Diabetes Centres
North Glasgow University Hospitals NHS Trust and
2Department of Medicine
University of Glasgow, Queen Elizabeth Building, Glasgow Royal Infirmary,
Glasgow G31 2ER
email jrp1s@clinmed.gla.ac.uk
note:
a)'older patients (>40)' should read 'greater or equal to' in symbol
form but this symbol was unable to be inserted into this email.
b) when 'Chi squared' appears twice this should be in symbol form but
again, this symbol was unable to be inserted into this email.
Competing interests: No competing interests
An Expertise Not Utilized
Dear Editor,
In response to Dr. Eysenbach's article, I feel that his research
assessing consumer web searching methods and web page validation
techniques, would have been further developed and substantiated if he had
utilized a librarian, and the librarian's knowledge of consumer searching.
Librarians are expert searchers, and provide a level of professionalism in
assessing search methods and outcomes that his article does not represent.
How can he state, "…search technique was often suboptimal…" or "Despite
the somewhat ineffective search strategies used…" when no examples
comparing "good" and "bad" search strategies are provided, or proof that
he or his co-author have the credentials or experience to evaluate search
strategies?
I commend Dr. Eysenbach on his awareness that research needs to be
conducted assessing Internet users and the health information they gleam.
I wonder what steps he plans to take in order to educate consumers in web
searching and evaluating the validity of the information they find. A
librarian has daily encounters with consumers seeking Internet
information. So why not collaborate with a librarian, to assess user
searching and instruct the users on the importance of knowing criteria to
evaluate Web pages.
Librarians have conducted research for years on user Web searching
techniques. When Dr. Eysenbach performed the background research to learn
what has been studied, did he evaluate the library literature? I have
included just a few of the many articles from librarian research that
would be relevant to a study such as his. Your "theme issue" demonstrates
an impressive awareness into a health concern where librarians can assist.
In fact, a librarian would have provided an expert perspective in nearly
all the research articles published in this issue.
Elizabeth M. LaRue
Head of Reference
Augustus C. Long Health Sciences Library,
Columbia University,
New York, NY 10032
1. Abate LE. Uncertainty in evaluating consumer health World Wide Web
sites [Thesis MSLS]. Univ of NC at Chapel Hill, 1998.
2. Connor E. Using search engines to locate Web resources on women's
health. Women's health on the Internet Haworth Press. United States,
2000:47-63 f.
3. Garrison S. Evaluating health Internet sites: a white paper's criteria.
Medical Reference Services Quarterly;17(3):41-7.
4. Khalil FEM. Consumer health information: a brief critique on
information needs and information seeking behaviour. Malaysian Journal of
Library & Information Science;6(2):83-99.
5. Littleton DA. A review of strategies for finding health information on
the World-Wide Web. Medical Reference Services Quarterly;17(2):51-5.
6. Lucas W, Topi H. Form and function: the impact of query term and
operator usage on Web search results. relevancy of search engine results.
Journal of the American Society for Information Science and
Technology;53(2):95-108.
7. Nicholas D, Huntington P, Williams PE. Evaluating metrics for comparing
the use of Web sites: a case study of two consumer health Web sites. NHS
Direct Online and SurgeryDoor. Journal of Information Science;28(1):63-75.
8. Silbajoris C. Health information needs and the Internet : a survey of
older adults [Thesis MSLS]. Univ of NC at Chapel Hill, 2000.
Competing interests:
None declared
Competing interests: No competing interests