Published criteria for evaluating health related web sites: reviewBMJ 1999; 318 doi: http://dx.doi.org/10.1136/bmj.318.7184.647 (Published 06 March 1999) Cite this as: BMJ 1999;318:647
- Paul Kim, research assistanta,
- Thomas R Eng, study director ()a,
- Mary Jo Deering, directora,
- Andrew Maxfield, research fellowb
- aHealth Communication and Telehealth, Office of Disease Prevention and Health Promotion, US Department of Health and Human Services, Washington DC, USA
- bNational Institute for Occupational Safety and Health, Centers for Disease Control and Prevention, Washington DC
- Correspondence to: Dr Eng
- Accepted 30 December 1998
Objective: To review published criteria for specifically evaluating health related information on the world wide web, and to identify areas of consensus.
Design: Search of world wide web sites and peer reviewed medical journals for explicit criteria for evaluating health related information on the web, using Medline and Lexis-Nexis databases, and the following internet search engines: Yahoo!, Excite, Altavista, Webcrawler, HotBot, Infoseek, Magellan Internet Guide, and Lycos. Criteria were extracted and grouped into categories.
Results: 29 published rating tools and journal articles were identified that had explicit criteria for assessing health related web sites. Of the 165 criteria extracted from these tools and articles, 132 (80%) were grouped under one of 12 specific categories and 33 (20%) were grouped as miscellaneous because they lacked specificity or were unique. The most frequently cited criteria were those dealing with content, design and aesthetics of site, disclosure of authors, sponsors, or developers, currency of information (includes frequency of update, freshness, maintenance of site), authority of source, ease of use, and accessibility and availability.
Conclusions: Results suggest that many authors agree on key criteria for evaluating health related web sites, and that efforts to develop consensus criteria may be helpful. The next step is to identify and assess a clear, simple set of consensus criteria that the general public can understand and use.
Many organisations and individuals have published criteria to evaluate health related information on the world wide web
A literature and world wide web search found that the most frequently cited criteria were those dealing with content, design and aesthetics of site, disclosure of authors, sponsors, or developers, currency of information, authority of source, and ease of use
Criteria related to confidentiality and privacy were only cited by one author
Consensus regarding critical criteria for evaluation of web based health information seems to be emerging
Our results indicate that many authors agree on key criteria for evaluating health related web sites, and that efforts to develop a set of key criteria may be helpful
The large volume of health information resources available on the internet has great potential to improve health,1–3 but it is increasingly difficult to discern which resources are accurate or appropriate for users.3–8 Because of the potential for harm from misleading and inaccurate health information,9–14 many organisations and individuals have published or implemented criteria for evaluating the appropriateness or quality of these resources. 15–16 Two published reviews of evaluation criteria for health related web sites did not present information on the range of criteria proposed by various authors, and included rating tools that were not developed exclusively for health related sites. 15–17 Our study reviews criteria currently proposed or employed specifically to evaluate health related web sites.
Databases and search engines
Between September 1997 and May 1998 we conducted a search of the web and peer reviewed medical journals for criteria for evaluating health related information on the web using Medline and Lexis-Nexis databases, and web search engines including Yahoo!, Excite, Altavista, Webcrawler, HotBot, Infoseek, Magellan Internet Guide, and Lycos. Medline searches (using PubMed) used variations of the following: “quality,” “Internet,” “World Wide Web,” “computer communication networks/standards,” “quality control,” and “medical Informatics/standards.” Searches with web search engines and Lexis-Nexis used “quality,” “health information,” “health,” and variations of “rating,” “ranking,” “evaluate,” “award,” and “assess.” Investigating references and hyperlinks from initial results gave additional resources. We ended the sampling period when searches produced similar results, and when previous search results became outdated.
We included criteria when they were explicit, specifically used for evaluating health related web sites, and published in a peer reviewed journal or publicly accessible web site. We also considered peer reviewed journals not indexed by Medline. We included resources framed as “guidelines” because there was little difference between them and other criteria, and the intent of the authors was similar. When subcriteria provided details about main criteria, we included only the main criteria to prevent overrepresenting that author's perspective. Criteria were extracted and sorted into similar groups according to their wording and description. When a criterion seemed to combine several concepts and could fit in multiple groups, we considered the first mentioned concept.
To examine the reproducibility of the criteria groupings, four independent, naive coders assigned 40 randomly selected criteria to the 13 criteria groups. Overall, the coders' assignment of criteria agreed with us 76% of the time. The agreement coefficient, indicating “per cent agreement above chance” was 0.74 or 74%.18
Twenty nine rating tools and articles—24 web sites and five journal articles—had explicit criteria for assessing health related web sites (table 1). Of the 165 criteria identified, 132 (80%) were grouped under 12 specific categories (table 2). Thirty three (20%) criteria that lacked specificity or were unique were categorised as “miscellaneous.” Frequently cited criteria included those dealing with content, design and aesthetics of site, and disclosure of authors, sponsors, or developers.
Not surprisingly, “content” of the site, which includes concepts of information quality and accuracy, was the most commonly cited criterion group. Design and aesthetics of the site and ease of use were the second and sixth most frequently cited groups respectively, indicating that authors highly value good quality application design and user interfaces. Disclosure of authors, sponsors, or developers had the third highest frequency, highlighting the need for users to be able to consider a site's content in the context of who created or financed the site. It was somewhat surprising that disclosure was not more commonly cited given recent reports about misleading health information and fraud on the internet. 91112 Most rating tools discriminated between content and the fourth most common criterion group, currency of information (includes frequency of update, freshness, maintenance of site), suggesting that currency of information is nearly as important as the information itself.
Criteria related to confidentiality and privacy of information were only cited by one author despite widespread interest in this issue.19 Some health related web sites are already collecting personal health information to “tailor” content, and as sites begin to integrate healthcare services and information, confidentiality and privacy safeguards will become increasingly important.19–21
Study limitations include the subjective variables around the scope of the criteria categories used. Testing of the category groupings, however, showed that they were reproducible by others. It is also possible that some authors used the same criteria terms to describe different concepts. Because subcriteria were not included, some concepts may not have been represented. Inherent limitations of web search engines and the dynamic nature of the web also prevented us from locating all existing published criteria.22 Nevertheless, our review located more sources of criteria specifically for health related sites than did previous reviews. 15 17
Given the evolving state of the internet, it may be difficult or even inappropriate to develop a static tool or system for assessing health related web sites. Our results suggest that many authors agree on key criteria, and that efforts to develop consensus criteria may be helpful. 6 16 23 25 The next step is to identify and assess a clear, simple set of consensus criteria that the general public can understand and use. Tools that integrate them need to be developed and validated, and their ultimate impact and effectiveness in assisting the public with health related decisions should be monitored to ensure that they remain useful.
We thank Farrokh Alemi and Anne Restino for their assistance and advice on this study. The views expressed in this paper are solely those of the authors and do not necessarily reflect those of the US Department of Health and Human Services.
Contributors: PK participated in data collection, analysis, and interpretation, and writing the paper. TRE formulated the study design, developed the core ideas, and participated in data analysis and interpretation, and writing the paper. MJD participated in the study design and interpretation, and edited the paper. AM participated in data analysis and interpretation, and edited the paper. PK and TRE will act as guarantors for the paper.
Funding: Internal funds of the US Department of Health and Human Services.
Competing interests: None declared.