BMJ 1999;318:647-649 ( 6 March )

Information in practice

Published criteria for evaluating health related web sites: review

Paul Kim, research assistant, a Thomas R Eng, study director, a Mary Jo Deering, director, a Andrew Maxfield, research fellow. b

a Health Communication and Telehealth, Office of Disease Prevention and Health Promotion, US Department of Health and Human Services, Washington DC, USA, b National Institute for Occupational Safety and Health, Centers for Disease Control and Prevention, Washington DC

Correspondence to: Dr Eng teng{at}osophs.dhhs.gov


    Abstract
Top
Abstract
Introduction
Methods
Results
Discussion
References

Objective: To review published criteria for specifically evaluating health related information on the world wide web, and to identify areas of consensus.
Design: Search of world wide web sites and peer reviewed medical journals for explicit criteria for evaluating health related information on the web, using Medline and Lexis-Nexis databases, and the following internet search engines: Yahoo!, Excite, Altavista, Webcrawler, HotBot, Infoseek, Magellan Internet Guide, and Lycos. Criteria were extracted and grouped into categories.
Results: 29 published rating tools and journal articles were identified that had explicit criteria for assessing health related web sites. Of the 165 criteria extracted from these tools and articles, 132 (80%) were grouped under one of 12 specific categories and 33 (20%) were grouped as miscellaneous because they lacked specificity or were unique. The most frequently cited criteria were those dealing with content, design and aesthetics of site, disclosure of authors, sponsors, or developers, currency of information (includes frequency of update, freshness, maintenance of site), authority of source, ease of use, and accessibility and availability.
Conclusions: Results suggest that many authors agree on key criteria for evaluating health related web sites, and that efforts to develop consensus criteria may be helpful. The next step is to identify and assess a clear, simple set of consensus criteria that the general public can understand and use.


Key messages

  • Many organisations and individuals have published criteria to evaluate health related information on the world wide web

  • A literature and world wide web search found that the most frequently cited criteria were those dealing with content, design and aesthetics of site, disclosure of authors, sponsors, or developers, currency of information, authority of source, and ease of use

  • Criteria related to confidentiality and privacy were only cited by one author

  • Consensus regarding critical criteria for evaluation of web based health information seems to be emerging

  • Our results indicate that many authors agree on key criteria for evaluating health related web sites, and that efforts to develop a set of key criteria may be helpful



    Introduction
Top
Abstract
Introduction
Methods
Results
Discussion
References

The large volume of health information resources available on the internet has great potential to improve health,1-3 but it is increasingly difficult to discern which resources are accurate or appropriate for users.3-8 Because of the potential for harm from misleading and inaccurate health information,9-14 many organisations and individuals have published or implemented criteria for evaluating the appropriateness or quality of these resources. 15 16 Two published reviews of evaluation criteria for health related web sites did not present information on the range of criteria proposed by various authors, and included rating tools that were not developed exclusively for health related sites. 15 17 Our study reviews criteria currently proposed or employed specifically to evaluate health related web sites.

    Methods
Top
Abstract
Introduction
Methods
Results
Discussion
References

Databases and search engines
Between September 1997 and May 1998 we conducted a search of the web and peer reviewed medical journals for criteria for evaluating health related information on the web using Medline and Lexis-Nexis databases, and web search engines including Yahoo!, Excite, Altavista, Webcrawler, HotBot, Infoseek, Magellan Internet Guide, and Lycos. Medline searches (using PubMed) used variations of the following: "quality," "Internet," "World Wide Web," "computer communication networks/standards," "quality control," and "medical Informatics/standards." Searches with web search engines and Lexis-Nexis used "quality," "health information," "health," and variations of "rating," "ranking," "evaluate," "award," and "assess." Investigating references and hyperlinks from initial results gave additional resources. We ended the sampling period when searches produced similar results, and when previous search results became outdated.

Criteria
We included criteria when they were explicit, specifically used for evaluating health related web sites, and published in a peer reviewed journal or publicly accessible web site. We also considered peer reviewed journals not indexed by Medline. We included resources framed as "guidelines" because there was little difference between them and other criteria, and the intent of the authors was similar. When subcriteria provided details about main criteria, we included only the main criteria to prevent overrepresenting that author's perspective. Criteria were extracted and sorted into similar groups according to their wording and description. When a criterion seemed to combine several concepts and could fit in multiple groups, we considered the first mentioned concept.

To examine the reproducibility of the criteria groupings, four independent, naive coders assigned 40 randomly selected criteria to the 13 criteria groups. Overall, the coders' assignment of criteria agreed with us 76% of the time. The agreement coefficient, indicating "per cent agreement above chance" was 0.74 or 74%.18

    Results
Top
Abstract
Introduction
Methods
Results
Discussion
References

Twenty nine rating tools and articles---24 web sites and five journal articles---had explicit criteria for assessing health related web sites (table 1). Of the 165 criteria identified, 132 (80%) were grouped under 12 specific categories (table 2). Thirty three (20%) criteria that lacked specificity or were unique were categorised as "miscellaneous." Frequently cited criteria included those dealing with content, design and aesthetics of site, and disclosure of authors, sponsors, or developers.


                              
View this table:
[in this window]
[in a new window]
 

Table 1. Rating tools and journal articles with explicit criteria for evaluation of health related web sites


                              
View this table:
[in this window]
[in a new window]
 

Table 2. Frequency of explicit criteria for evaluation of health related web sites by criteria groups*



    Discussion
Top
Abstract
Introduction
Methods
Results
Discussion
References

Not surprisingly, "content" of the site, which includes concepts of information quality and accuracy, was the most commonly cited criterion group. Design and aesthetics of the site and ease of use were the second and sixth most frequently cited groups respectively, indicating that authors highly value good quality application design and user interfaces. Disclosure of authors, sponsors, or developers had the third highest frequency, highlighting the need for users to be able to consider a site's content in the context of who created or financed the site. It was somewhat surprising that disclosure was not more commonly cited given recent reports about misleading health information and fraud on the internet. 9 11 12 Most rating tools discriminated between content and the fourth most common criterion group, currency of information (includes frequency of update, freshness, maintenance of site), suggesting that currency of information is nearly as important as the information itself.

Criteria related to confidentiality and privacy of information were only cited by one author despite widespread interest in this issue.19 Some health related web sites are already collecting personal health information to "tailor" content, and as sites begin to integrate healthcare services and information, confidentiality and privacy safeguards will become increasingly important.19-21

Study limitations
Study limitations include the subjective variables around the scope of the criteria categories used. Testing of the category groupings, however, showed that they were reproducible by others. It is also possible that some authors used the same criteria terms to describe different concepts. Because subcriteria were not included, some concepts may not have been represented. Inherent limitations of web search engines and the dynamic nature of the web also prevented us from locating all existing published criteria.22 Nevertheless, our review located more sources of criteria specifically for health related sites than did previous reviews. 15 17

Conclusion
Given the evolving state of the internet, it may be difficult or even inappropriate to develop a static tool or system for assessing health related web sites. Our results suggest that many authors agree on key criteria, and that efforts to develop consensus criteria may be helpful. 6 16 23-25 The next step is to identify and assess a clear, simple set of consensus criteria that the general public can understand and use. Tools that integrate them need to be developed and validated, and their ultimate impact and effectiveness in assisting the public with health related decisions should be monitored to ensure that they remain useful.

    Acknowledgments

We thank Farrokh Alemi and Anne Restino for their assistance and advice on this study. The views expressed in this paper are solely those of the authors and do not necessarily reflect those of the US Department of Health and Human Services.

Contributors: PK participated in data collection, analysis, and interpretation, and writing the paper. TRE formulated the study design, developed the core ideas, and participated in data analysis and interpretation, and writing the paper. MJD participated in the study design and interpretation, and edited the paper. AM participated in data analysis and interpretation, and edited the paper. PK and TRE will act as guarantors for the paper.

Funding: Internal funds of the US Department of Health and Human Services.

Competing interests: None declared.

    References
Top
Abstract
Introduction
Methods
Results
Discussion
References

  1. Government Accounting Office. Consumer health informatics. Emerging issues. Publication Government Accounting Office/Accounting and Information Management Division-96-86, July 1996.
  2. Robinson TN, Patrick K, Eng TR, Gustafson D, for the Science Panel on Interactive Communication and Health. An evidence-based approach to interactive health communication: a challenge to medicine in the Information Age. JAMA 1998; 280: 1264-1269[Abstract/Free Full Text].
  3. Eng TR, Maxfield A, Patrick K, Deering MJ, Ratzan S, Gustafson D. Access to health information and support: a public highway or a private road? JAMA 1998; 280: 1371-1375[Abstract/Free Full Text].
  4. Coiera E. The internet's challenge to health care provision. BMJ 1996; 312: 3-4[Free Full Text].
  5. Anon. The web of information inequality [editorial]. Lancet 1997; 349: 1781[Medline].
  6. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the internet. Caveant lector et viewor---let the reader and buyer beware. JAMA 1997; 277: 1244-1245[Medline].
  7. Sonnenberg FA. Health information on the internet. Opportunities and pitfalls [editorial]. Arch Intern Med 1997; 157: 151-152[Medline].
  8. Wyatt JC. Commentary: measuring quality and impact of the world wide web. BMJ 1997; 314: 1879-1881[Free Full Text].
  9. Federal Trade Commission. North American Health Claim Surf Day targets Internet ads. Hundreds of e-mail messages sent. Press release, Nov. 5, 1997. Accessed May 12, 1998. http://www.ftc.gov/opa/9711/hlthsurf.htm
  10. Impicciatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the world wide web: systematic survey of advice on managing fever in children at home. BMJ 1997; 314: 1875-1879[Abstract/Free Full Text].
  11. Food and Drug Administration. FDA warns consumers on dangerous products promoted on the internet. FDA Talk Paper T97-26, June 17, 1997.
  12. Bower H. Internet sees growth of unverified health claims. BMJ 1996; 313: 497[Free Full Text].
  13. Micke MM. The case of hallucinogenic plants and the internet. J Sch Health 1996; 66: 277-280[Medline].
  14. Weisbord SD, Soule JB, Kimmel PL. Poison on line---acute renal failure caused by oil of wormwood purchased through the internet. N Engl J Med 1997; 337: 825-827[Free Full Text].
  15. Jadad AR, Gagliardi A. Rating health information on the internet. Navigating to knowledge or to Babel? JAMA 1998; 279: 611-614[Abstract/Free Full Text].
  16. Pealer LN, Dorman SM. Evaluating health-related web sites. J Sch Health 1997; 67: 232-235[Medline].
  17. Murray PJ, Rizzolo MA. Web site reviews and evaluations. Nurs Stand Online 1997 Jul 30;11(45). Accessed January 29, 1998. http://www.nursing-standard.co.uk/vol11-45/ol-art.htm
  18. Krippendorf K. Content analysis: an introduction to its methodology. Beverly Hills, CA: Sage , 1980.
  19. Bowen JW, Klimczak JC, Ruiz M, Barnes M. Design of access control methods for protecting the confidentiality of patient information in networked systems. Proceedings of the American Medical Informatics Association annual fall symposium 1997:46-50.
  20. National Research Council, Computer Science and Telecommunications Board (US). For the record: protecting electronic health information. Washington: National Academy Press , 1997.
  21. Patrick K, Robinson TN, Alemi F, Eng TR, for the Science Panel on Interactive Communication and Health. Policy issues relevant to the evaluation of interactive health communication applications. Am J Prev Med 1999; 16: 35-42[Medline].
  22. Lawrence S, Giles CL. Searching the world wide web. Science 1998; 280: 98-100[Abstract/Free Full Text].
  23. British Healthcare Internet Association. Quality standards for medical publishing on the web. Accessed May 26, 1998. http://www.bhia.org/public/reference/recommendations/medpubstandards.htm
  24. Health On the Net Foundation. HON code of conduct for medical and health web sites. Accessed January 27, 1998. http://www.hon.ch/HONcode/Conduct.html
  25. Health Information Technology Institute, Mitretek Systems. Criteria for assessing the quality of health information on the internet. Accessed January 27, 1998. http://www.mitretek.org/hiti/showcase/index.html

(Accepted 30 December 1998)

© BMJ 1999

Related Article

Consensus on criteria for evaluating health related web sites is emerging
BMJ 1999 318: 0. [Full Text]

This article has been cited by other articles:

  • Marriott, J. V., Stec, P., El-Toukhy, T., Khalaf, Y., Braude, P., Coomarasamy, A. (2008). Infertility information on the World Wide Web: a cross-sectional survey of quality of infertility information on the internet in the UK. Hum Reprod 0: den072v1-6 [Abstract] [Full text]  
  • Datta, A. K, Selman, T. J, Kwok, T., Tang, T., Khan, K. S (2008). Quality of information accompanying on-line marketing of home diagnostic tests. JRSM 101: 34-38 [Abstract] [Full text]  
  • Harrison, S., Barlow, J., Williams, G. (2007). The content and interactivity of health support group websites. Health Education Journal 66: 371-381 [Abstract]  
  • Fragoulis, K. N., Vardakas, K. Z., Falagas, M. E. (2007). Open access World Wide Web resources on urogenital infections. Nephrol Dial Transplant 22: 3046-3047 [Abstract] [Full text]  
  • Harland, J., Bath, P. (2007). Assessing the quality of websites providing information on multiple sclerosis: evaluating tools and comparing sites. Health Informatics Journal 13: 207-221 [Abstract]  
  • St Leger, L. (2006). Communication technologies and health promotion: opportunities and challenges. HEALTH PROMOT INT 21: 169-171 [Full text]  
  • Papadaki, A, Scott, J A (2006). Process evaluation of an innovative healthy eating website promoting the Mediterranean diet. Health Educ Res 21: 206-218 [Abstract] [Full text]  
  • Hargrave, D. R., Hargrave, U. A., Bouffet, E. (2006). Quality of health information on the Internet in pediatric neuro-oncology. Neuro Oncol 8: 175-182 [Abstract] [Full text]  
  • Shaikh, U., Scott, B. J. (2005). Extent, Accuracy, and Credibility of Breastfeeding Information on the Internet. J Hum Lact 21: 175-183 [Abstract]  
  • Martins, E N, Morse, L S (2005). Evaluation of internet websites about retinopathy of prematurity patient education. Br. J. Ophthalmol. 89: 565-568 [Abstract] [Full text]  
  • Smith, L., Heys, M. (2005). How to evaluate your website. BMJ CAREER FOCUS 330: 178-178 [Full text]  
  • McCray, A. T. (2005). Promoting Health Literacy. J. Am. Med. Inform. Assoc. 12: 152-163 [Abstract] [Full text]  
  • Charnock, D., Shepperd, S. (2004). Learning to DISCERN online: applying an appraisal tool to health websites in a workshop setting. Health Educ Res 19: 440-446 [Abstract] [Full text]  
  • Benigeri, M., Pluye, P. (2003). Shortcomings of health information on the Internet. HEALTH PROMOT INT 18: 381-386 [Abstract] [Full text]  
  • Sheehan, N. L, Kelly, D. V, Tseng, A. L, van Heeswijk, R. P., Beique, L. C, Hughes, C. A (2003). Evaluation of HIV Drug Interaction Web Sites. The Annals of Pharmacotherapy 37: 1577-1586 [Abstract] [Full text]  
  • Theodosiou, L., Green, J. (2003). Emerging challenges in using health information from the internet. Adv. Psychiatr. Treat. 9: 387-396 [Abstract] [Full text]  
  • Bath, P. A., Bouchier, H. (2003). Development and Application of a Tool Designed to Evaluate Web Sites Providing Information on Alzheimer's Disease. Journal of Information Science 29: 279-297 [Abstract]  
  • Cheh, J. A., Ribisl, K. M., Wildemuth, B. M. (2003). An Assessment of the Quality and Usability of Smoking Cessation Information on the Internet. Health Promot Pract 4: 278-287 [Abstract]  
  • Bouchier, H., Bath, P. A. (2003). Evaluation of Websites that Provide Information on Alzheimer's Disease. Health Informatics Journal 9: 17-31 [Abstract]  
  • Al-Shahi, R, Sadler, M, Rees, G, Bateman, D (2002). The internet. J. Neurol. Neurosurg. Psychiatry 73: 619-628 [Abstract] [Full text]  
  • Bensley, R. J., Lewis, J. B. (2002). Analysis of Internet-Based Health Assessments. Health Promot Pract 3: 463-476 [Abstract]  
  • Howitt, A., Clement, S., de Lusignan, S., Thiru, K., Goodwin, D., Wells, S. (2002). An evaluation of general practice websites in the UK. Fam Pract 19: 547-556 [Abstract] [Full text]  
  • Fallis, D., Fricke, M. (2002). In reply:--. J. Am. Med. Inform. Assoc. 9: 403-404 [Full text]  
  • Wolfe, R. M., Sharp, L. K., Lipsky, M. S. (2002). Content and Design Attributes of Antivaccination Web Sites. JAMA 287: 3245-3248 [Abstract] [Full text]  
  • Eysenbach, G., Powell, J., Kuss, O., Sa, E.-R. (2002). Empirical Studies Assessing the Quality of Health Information for Consumers on the World Wide Web: A Systematic Review. JAMA 287: 2691-2700 [Abstract] [Full text]  
  • Croft, D. R., Peterson, M. W. (2002). An Evaluation of the Quality and Contents of Asthma Education on the World Wide Web*. Chest 121: 1301-1307 [Abstract] [Full text]  
  • Kunst, H., Groot, D., Latthe, P. M, Latthe, M., Khan, K. S (2002). Accuracy of information on apparently credible websites: survey of five common health topics. BMJ 324: 581-582 [Full text]  
  • D'Alessandro, D., Kingsley, P. (2002). Creating a Pediatric Digital Library for Pediatric Health Care Providers and Families: Using Literature and Data to Define Common Pediatric Problems. J. Am. Med. Inform. Assoc. 9: 161-170 [Abstract] [Full text]  
  • Clement, W A, Wilson, S, Bingham, B J (2002). A guide to creating your own patient-oriented website. JRSM 95: 64-67 [Full text]  
  • Fallis, D., Fricke, M. (2002). Indicators of Accuracy of Consumer Health Information on the Internet: A Study of Indicators Relating to Information for Managing Fever in Children in the Home. J. Am. Med. Inform. Assoc. 9: 73-79 [Abstract] [Full text]  
  • VandeKieft, G. K., Ogle, K. S., Wang, T. (2001). Internet resources for end-of-life care. AM J HOSP PALLIAT CARE 18: 313-320 [Abstract]  
  • Berland, G. K., Elliott, M. N., Morales, L. S., Algazy, J. I., Kravitz, R. L., Broder, M. S., Kanouse, D. E., Munoz, J. A., Puyol, J.-A., Lara, M., Watkins, K. E., Yang, H., McGlynn, E. A. (2001). Health Information on the Internet: Accessibility, Quality, and Readability in English and Spanish. JAMA 285: 2612-2621 [Abstract] [Full text]  
  • Coomarasamy, A., Latthe, P., Papaioannou, S., Publicover, M., Gee, H., Khan, K. S (2001). Critical appraisal in clinical practice: sometimes irrelevant, occasionally invalid. JRSM 94: 573-577 [Abstract] [Full text]  
  • Smith, M., Gertz, E., Alvarez, S., Lurie, P. (2000). The Content and Accessibility of Sex Education Information on the Internet. Health Educ Behav 27: 684-694 [Abstract]  
  • Edejer, T. T.-T. (2000). Disseminating health information in developing countries: the role of the internet. BMJ 321: 797-800 [Full text]  
  • ARMSTRONG, R. (2000). http://www.ilar.org A source of information for rheumatologists, allied health professionals, medical students and the general public. Ann Rheum Dis 59: 241-242 [Full text]  
  • Jadad, A. R., Haynes, R. B., Hunt, D., Browman, G. P. (2000). The Internet and evidence-based decision-making: a needed synergy for efficient knowledge management in health care. CMAJ 162: 362-365 [Full text]  
  • Jadad, A. R (1999). Promoting partnerships: challenges for the internet age. BMJ 319: 761-764 [Full text]  
  • Shepperd, S., Charnock, D., Gann, B. (1999). Helping patients access high quality health information. BMJ 319: 764-766 [Full text]  
  • Tramer, M. R. (1999). How Can We Cope with the Internet?. Anesth. Analg. 89: 271-271 [Full text]  

Rapid Responses:

Read all Rapid Responses

A wider application than just to doctors
Peter Murray
bmj.com, 11 Mar 1999 [Full text]
Internet evaluation tools better aimed at professionals
Michelle Gillies
bmj.com, 7 Apr 1999 [Full text]

Student BMJ

Risk of surgery for inflammatory bowel disease: record linkage studies

What can you learn from this BMJ paper? Read Leanne Tite's Paper+

www.student.bmj.com

Listen to the latest BMJ Interview