BMJ 2002;324:569-573 ( 9 March )

Papers

Examination of instruments used to rate quality of health information on the internet: chronicle of a voyage with an unclear destination

Anna Gagliardi, senior research associate a, Alejandro R Jadad, professor b. 

a See Education and debate p 606 Graduate Department of Health Policy, Management and Evaluation, Faculty of Medicine, University of Toronto, Toronto, ON, Canada, b Departments of Health Policy, Management and Evaluation, and Anaesthesia, University Health Network, University of Toronto, Toronto, ON, Canada

Correspondence to: A R Jadad, Director, Centre for Global eHealth Innovation, University Health Network, Toronto General Hospital, Fraser Elliott Building, 4th Floor, 190 Elizabeth Street, Toronto, ON M5G 2C4, Canada ajadad{at}uhnres.utoronto.ca


    Abstract
Top
Abstract
Introduction
Methods
Results
Discussion
References

Objective: This study updates work published in 1998, which found that of 47 rating instruments appearing on websites offering health information, 14 described how they were developed, five provided instructions for use, and none reported the interobserver reliability and construct validity of the measurements.
Design: All rating instrument sites noted in the original study were visited to ascertain whether they were still operating. New rating instruments were identified by duplicating and enhancing the comprehensive search of the internet and the medical and information science literature used in the previous study. Eligible instruments were evaluated as in the original study.
Results: 98 instruments used to assess the quality of websites in the past five years were identified. Many of the rating instruments identified in the original study were no longer available. Of 51 newly identified rating instruments, only five provided some information by which they could be evaluated. As with the six sites identified in the original study that remained available, none of these five instruments seemed to have been validated.
Conclusions: Many incompletely developed rating instruments continue to appear on websites providing health information, even when the organisations that gave rise to those instruments no longer exist. Many researchers, organisations, and website developers are exploring alternative ways of helping people to find and use high quality information available on the internet. Whether they are needed or sustainable and whether they make a difference remain to be shown.

What is already known on this topic
The rapid growth of healthcare websites in the 1990s was accompanied by initiatives to rate their quality, including award-like symbols on websites

A systematic review of the reliability and validity of such rating instruments, published in 1998, showed that they were incompletely developed

What this study adds
Few of the rating instruments identified in 1998 remain functional; 51 new instruments were identified

Of the 51 newly identified instruments, 11 were not functional, 35 were available but provided no information, and five provided information but were not validated

Many researchers, organisations, and website developers are exploring alternative ways of helping people to find high quality information on the internet




    Introduction
Top
Abstract
Introduction
Methods
Results
Discussion
References

The quality of health information on the internet became a subject of interest to healthcare professionals, information specialists, and consumers of health care in the mid-1990s. Along with the rapid growth of healthcare websites came a number of initiatives, both academic and commercial, that generated criteria by which to ensure, judge, or denote the quality of websites offering health information. Some of these rating instruments took the form of logos resembling "awards" or "seals of approval" and appeared prominently on the websites on which they were bestowed.

In 1997 we undertook a review of "award-like" internet rating instruments in an effort to assess their reliability and validity.1 We hypothesised that if the rating instruments were flawed they might influence healthcare providers or consumers relying on them as indicators of accurate information. Instruments were eligible for review if they had been used at least once to categorise a website offering health information and revealed the rating criteria by which they did so. The rating instruments were evaluated according to, firstly, a system for judging the rigour of the development of tools to assess the quality of randomised controlled trials2 and, secondly, whether their criteria included three indicators suggested as appropriate for judging the quality of website content. 3 4 These indicators were authorship (information about authors and their contributions, affiliations, and relevant credentials), attribution (listing of references or sources of content), and disclosure (a description of website ownership, sponsorship, underwriting, commercial funding arrangements, or potential conflicts of interest). These criteria were selected for use in the original study because they could be rated objectively.

Our original study found that of 47 rating instruments identified, 14 described how they were developed, five provided instructions for use, and none reported the interobserver reliability and construct validity of the measurements. The review showed that many incompletely developed instruments were being used to evaluate or draw attention to health information on the internet.

The purpose of this study is to update the previous review of award-like rating instruments for the evaluation of websites providing health information and to describe any changes that may have taken place in the development of websites offering health information to practitioners and consumers with respect to the quality of their content.


    Methods
Top
Abstract
Introduction
Methods
Results
Discussion
References

We visited the websites describing each of the rating instruments noted in the original study to ascertain whether they were still operating. If internet service was disrupted for technical reasons or if sites were not available on first visit, we attempted a connection on one further occasion.

The search strategies, inclusion and exclusion criteria, and techniques for data extraction were similar to those used in the original review.1 We used the following sources to identify new rating instruments:

  • A search to 7 September 2001 of Medline, CINAHL, and HealthSTAR (from December 1997) using [(top or rat: or rank: or best) and (internet or web) and (quality or reliab: or valid:)]

  • A search of the databases Information Science Abstracts, Library and Information Science Abstracts (1995 to September 2001), and Library Literature (1996 to September 2001) using [(rat: or rank: or top or best) and (internet or web or site) and (health:)]

  • A search of the American Medical Informatics Association's 1998, 1999, 2000, and 2001 annual symposium programmes (www.amia.org) for mention of health information on the internet


                              
View this table:
[in this window]
[in a new window]
 

Table 1. Summary of criteria for rating instruments

  • A search of the Journal of Medical Internet Research (September 1999 to September 2001) for mention of evaluations of the quality of health information on the internet (www.jmir.org)

  • A search of the online archive of the magazine Internet World (www.internetworld.com) (January 2000 to September 2001) for mention of health information on the internet.

We also reviewed relevant articles referenced in identified studies and links available on identified websites. We did not search the discussion list Public Communication of Science and Technology, which was consulted in the original study.

We stopped searching for rating instruments on 22 September 2001. Rating instruments were eligible for inclusion in the review if it was possible to link from their award-like symbol to an available website describing the criteria used by an individual or organisation to judge the quality of websites on which the award was bestowed. We excluded rating instruments from review if they were used only to rate sites offering non-health information or did not provide any description of their rating criteria. In contrast to the initial study, we did not contact the developers of rating instruments to request information about their criteria if it was not publicly available on their website.

We identified the website, group, or organisation that developed each eligible rating instrument, along with its web address. The two authors independently evaluated each rating instrument according to its validity (number of items in the instrument, availability of rating instructions, information on the development of rating criteria, and evaluation of interobserver reliability) and incorporation of the proposed criteria for evaluation of internet sites: authorship, attribution, and disclosure.2-4


    Results
Top
Abstract
Introduction
Methods
Results
Discussion
References

Fourteen rating instruments identified in the original study provided a description of their rating criteria and were therefore eligible for review. Six of these continued to function. Of the remaining eight instruments, four were no longer in operation and four had converted to a directory format. Table 1 summarises the review of the six functioning instruments. Our evaluation of one of these instruments, OncoLink's editors' choice awards, differed from that in the original study because the organisation does not provide information about the instrument on its website.

Of the 33 rating instruments identified in the original study that were not eligible for review, three continued to function. These were Best Medical Resources on the Web (priory.com/other.htm), Dr Webster's website of the day (drWebster.com), and HealthSeek quality site award (healthseek.com). None of these rating instruments revealed its rating criteria, and they therefore remained ineligible for review. Of the remaining rating instrument websites, 10 were no longer in operation, five had been subsumed by or merged with another organisation and had a different name or purpose, and 15 still offered a website but did not function as a rating instrument.

We newly identified 51 rating instruments. Eleven of these were identified as award-like symbols on a website offering health information, but the website of the organisation from which they originated was no longer operating (table 2). Of the remaining 40 rating instruments, 35 were associated with an active website but did not reveal the criteria by which they judge websites and were ineligible for evaluation (table 3). Five award sites discussed their evaluation criteria and were assessed (table 1). Although three of these five rating instruments exhibited one or more of the characteristics of authorship, attribution, and disclosure, none reported on the reliability and validity of the measurements or provided instructions on how to obtain the ratings.


                              
View this table:
[in this window]
[in a new window]
 

Table 2. Newly identified award sites not available


                              
View this table:
[in this window]
[in a new window]
 

Table 3. Newly identified available award sites not eligible for review




    Discussion
Top
Abstract
Introduction
Methods
Results
Discussion
References

During the past five years, we have identified a total of 98 different rating instruments that have been used to assess the quality of websites. Many of the rating instruments identified in the original study were no longer available. Fifty one additional rating instruments have been developed since 1997, and many of these had also stopped functioning. Of 51 newly identified rating instruments, only five provided some information by which they could be evaluated. As with the six rating instrument sites identified in the original study that remained available, none of these seems to have been validated. Many incompletely developed rating instruments continue to appear on websites providing health information, even when the organisations that gave rise to them no longer exist. Surprisingly, many of these rating instruments, of questionable utility and without association to an operable entity, are featured on the US Department of Health and Human Services Healthfinder website (www.healthfinder.gov/aboutus/awards.htm), which uses a detailed and rigorous selection process for the development of its own content.

Our initial questions remain unanswered. Is it desirable or necessary to assess the quality of health information on the internet? If so, is it an achievable goal given that quality is a construct for which we have no gold standard? Some effort has been made to identify whether the presence of rating instrument awards influences consumers of health information,5 but whether validated rating instruments would have an impact on the competence, performance, behaviour, and health outcomes of those who use them remains unclear.

Our search of the literature and the internet revealed that a large number of researchers, organisations, and website developers are exploring alternative ways to help people find and use high quality information available on the internet. Many reviews of healthcare information on the internet have been conducted, overall and for specific diseases or conditions.6-12 Examination of over 90 reviews concluded that the validity of health information available on websites is highly variable across different diseases and populations, and is in many cases potentially misleading or harmful (G Eysenbach, personal communication, 2001). Several organisations, including government and non-profit entities, have developed criteria by which to organise and identify valid health information (table 4). Other groups, such as the OMNI Advisory Group for Evaluation Criteria (omni.ac.uk) and the Collaboration for Critical Appraisal of Information on the Net (www.medcertain.org), are refining technical mechanisms by which users of the internet can easily locate quality health information in a transparent manner based on evaluative meta-information labelling and indexing.13-15 The impact of these efforts remains unclear.


                              
View this table:
[in this window]
[in a new window]
 

Table 4. Initiatives to organise and identify valid health information on the internet

More recently, a European project recommended the accreditation of healthcare related software, telemedicine, and internet sites.16 They suggested a mechanism similar to the marking of electrical goods for software, that national regulatory bodies should be identified for telemedicine, and that a European certification of integrity scheme should be developed for websites. Citing the many impediments to voluntary quality assurance for websites, the authors suggest the development of criteria, modifiable according to the needs of special interest groups, that would be used by accredited agencies to self label conforming websites (not only those offering health information) with a EuroSeal. Monitoring of integrity would be ongoing through cryptographic techniques.

In conclusion, our updated study shows that award systems based on non-validated rating instruments continue to be produced but that most stop functioning soon after their release. Alternative strategies are now flourishing, and whether they are valid, needed, or sustainable and whether they make a difference is the subject of further research.

    Acknowledgments

   Contributors: AG conducted the searches, extracted relevant data, evaluated eligible instruments, and drafted the manuscript. ARJ developed the idea for the original study, independently evaluated eligible instruments, edited the manuscript, and is guarantor for this paper.

    Footnotes

Funding: ARJ was supported by funds from the University Health Network, the Rose Family Chair in Supportive Care, and a Premier's Research Excellence Award from the Ministry of Energy, Science and Technology of Ontario.

Competing interests: None declared.


    References
Top
Abstract
Introduction
Methods
Results
Discussion
References

1. Jadad AR, Gagliardi A. Rating health information on the internet: navigating to knowledge or to Babel? JAMA 1998; 279: 611-614[Abstract/Free Full Text].
2. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography. Control Clin Trials 1995; 16: 62-73[ISI][Medline].
3. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling and assuring the quality of medical information on the internet. JAMA 1997; 277: 1244-1245[CrossRef][ISI][Medline].
4. Wyatt JC. Measuring quality and impact of the world wide web [commentary]. BMJ 1997; 314: 1879-1881[Free Full Text].
5. Shon J, Marshall J, Musen MA. The impact of displayed awards on the credibility and retention of web site information. Proc AMIA Symp 2000:794-8.
6. Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, et al. Health information on the internet: accessibility, quality, and readability in English and Spanish. JAMA 2001; 285: 2612-2621[Abstract/Free Full Text].
7. Li L, Irvin E, Guzman J, Bombardier C. Surfing for back pain patients: the nature and quality of back pain information on the internet. Spine 2001; 26: 545-547[CrossRef][ISI][Medline].
8. Suarez-Almazor ME, Kendall CJ, Dorgan M. Surfing the net---information on the world wide web for persons with arthritis: patient empowerment or patient deceit? J Rheumatol 2001; 28: 185-191[ISI][Medline].
9. Impiccatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the world wide web: systemic survey of advice on managing fever in children at home. BMJ 1997; 314: 1875-1879[Abstract/Free Full Text].
10. Griffiths KM, Christensen H. Quality of web based information on treatment of depression: cross sectional survey. BMJ 2000; 321: 1511-1515[Abstract/Free Full Text].
11. Abbott VP. Web page quality: can we measure it and what do we find? A report of exploratory findings. J Public Health Med 2000; 22: 191-197[Abstract/Free Full Text].
12. Tamm EP, Raval BK, Huynh PT. Evaluation of the quality of self-education mammography material available for patients on the internet. Acad Radiol 2000; 7: 137-141[CrossRef][ISI][Medline].
13. Eysenbach G, Diepgen TL. Towards quality management of medical information on the internet: evaluation, labelling, and filtering of information. BMJ 1998; 317: 1496-1500[Free Full Text].
14. Eysenbach G, Diepgen TL. Labeling and filtering of medical information on the internet. Methods Inf Med 1999; 38: 80-88[ISI][Medline].
15. Price SL, Hersh WR. Filtering web pages for quality indicators: an empirical approach to finding high quality consumer health information on the world wide web. Proc AMIA Symp 1999:911-5.
16. Rigby M, Forsstrom J, Roberts R, Wyatt J. Verifying quality and safety in health informatics services. BMJ 2001; 323: 552-556[Free Full Text].

© BMJ 2002
Add to CiteULike CiteULike   Add to Complore Complore   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati    What's this?

Related Articles

Developing a quality criteria framework for patient decision aids: online international Delphi consensus process
Glyn Elwyn, Annette O'Connor, Dawn Stacey, Robert Volk, Adrian Edwards, Angela Coulter, Richard Thomson, Alexandra Barratt, Michael Barry, Steven Bernstein, Phyllis Butow, Aileen Clarke, Vikki Entwistle, Deb Feldman-Stewart, Margaret Holmes-Rovner, Hilary Llewellyn-Thomas, Nora Moumjid, Al Mulley, Cornelia Ruland, Karen Sepucha, Alan Sykes, Tim Whelan The International Patient Decision Aids Standards (IPDAS) Collaboration
BMJ 2006 333: 417. [Abstract] [Full Text] [PDF]

Are virtual communities good for our health?
Alejandro R Jadad, Murray W Enkin, Sholom Glouberman, Philip Groff, and Anita Stern
BMJ 2006 332: 925-926. [Extract] [Full Text] [PDF]

Is a consultation needed?
Frank Sullivan and Jeremy C Wyatt
BMJ 2005 331: 625-627. [Extract] [Full Text] [PDF]

Unvalidated tools to rate web health information are proliferating
BMJ 2002 324: 0. [Full Text] [PDF]

The invention of talk
BMJ 2002 324: 0. [Full Text] [PDF]

The quality of health information on the internet
Gretchen P Purcell, Petra Wilson, and Tony Delamothe
BMJ 2002 324: 557-558. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

  • Bath, P. A. (2008). Health informatics: current issues and challenges. Journal of Information Science 34: 501-518 [Abstract]  
  • Akram, G, Thomson, A H, Boyter, A C, Morton, M J S (2008). Characterisation and evaluation of UK websites on attention deficit hyperactivity disorder. Arch. Dis. Child. 93: 695-700 [Abstract] [Full text]  
  • Keselman, A., Logan, R., Smith, C. A., Leroy, G., Zeng-Treitler, Q. (2008). Developing Informatics Tools and Strategies for Consumer-centered Health Communication. J. Am. Med. Inform. Assoc. 15: 473-483 [Abstract] [Full text]  
  • Ferreira-Lay, P., Miller, S. (2008). The quality of internet information on depression for lay people. Psychiatr. Bull. 32: 170-173 [Abstract] [Full text]  
  • Khechine, H., Pascot, D., Premont, P. (2008). Use of health-related information from the Internet by English-speaking patients. Health Informatics Journal 14: 17-28 [Abstract]  
  • Datta, A. K, Selman, T. J, Kwok, T., Tang, T., Khan, K. S (2008). Quality of information accompanying on-line marketing of home diagnostic tests. JRSM 101: 34-38 [Abstract] [Full text]  
  • Porter, A., Edirippulige, S. (2007). Parents of Deaf Children Seeking Hearing Loss-Related Information on the Internet: the Australian Experience. J Deaf Stud Deaf Educ 12: 518-529 [Abstract] [Full text]  
  • Elwyn, G., O'Connor, A., Stacey, D., Volk, R., Edwards, A., Coulter, A., Thomson, R., Barratt, A., Barry, M., Bernstein, S., Butow, P., Clarke, A., Entwistle, V., Feldman-Stewart, D., Holmes-Rovner, M., Llewellyn-Thomas, H., Moumjid, N., Mulley, A., Ruland, C., Sepucha, K., Sykes, A., Whelan, T., The International Patient Decision Aids Standards, (2006). Developing a quality criteria framework for patient decision aids: online international Delphi consensus process. BMJ 333: 417- [Abstract] [Full text]  
  • Jadad, A. R, Enkin, M. W, Glouberman, S., Groff, P., Stern, A. (2006). Are virtual communities good for our health?. BMJ 332: 925-926 [Full text]  
  • Hargrave, D. R., Hargrave, U. A., Bouffet, E. (2006). Quality of health information on the Internet in pediatric neuro-oncology. Neuro Oncol 8: 175-182 [Abstract] [Full text]  
  • Schwartz, K. L., Roe, T., Northrup, J., Meza, J., Seifeldin, R., Neale, A. V. (2006). Family Medicine Patients' Use of the Internet for Health Information: A MetroNet Study. J Am Board Fam Med 19: 39-45 [Abstract] [Full text]  
  • Ritterband, L. M., Borowitz, S., Cox, D. J., Kovatchev, B., Walker, L. S., Lucas, V., Sutphen, J. (2005). Using the Internet to Provide Information Prescriptions. Pediatrics 116: e643-e647 [Abstract] [Full text]  
  • Sullivan, F., Wyatt, J. C (2005). Is a consultation needed?. BMJ 331: 625-627 [Full text]  
  • Broom, A. (2005). Medical specialists' accounts of the impact of the Internet on the doctor/patient relationship. Health (London) 9: 319-338 [Abstract]  
  • Ansani, N. T., Vogt, M., Henderson, B. A. F., Mckaveney, T. P., Weber, R. J., Smith, R. B., Burda, M., Kwoh, C. K., Osial, T. A., Starz, T. (2005). Quality of arthritis information on the Internet. Am J Health Syst Pharm 62: 1184-1189 [Abstract] [Full text]  
  • Purvis, J. M., Alexander, A. H., Einhorn, T. A., Griffin, L. Y. (2005). Symposium Evaluating the Flood of Orthopaedic Media and Marketing Information. JBJS 87: 1392-1398 [Full text]  
  • McCray, A. T. (2005). Promoting Health Literacy. J. Am. Med. Inform. Assoc. 12: 152-163 [Abstract] [Full text]  
  • SPALLEK, H. (2004). A resource guide for practice development through technology. Journal of the American Dental Association 135: 38S-44S [Full text]  
  • Charnock, D., Shepperd, S. (2004). Learning to DISCERN online: applying an appraisal tool to health websites in a workshop setting. Health Educ Res 19: 440-446 [Abstract] [Full text]  
  • Anselmo, M. A., Lash, K. M., Stieb, E. S., Haver, K. E. (2004). Cystic Fibrosis on the Internet: A Survey of Site Adherence to AMA Guidelines. Pediatrics 114: 100-103 [Abstract] [Full text]  
  • Kim, S., Mouradian, W. E., Leggott, P. J., Schaad, D. C., Shaul, C. (2004). Implications for Designing Online Oral Health Resources: A Review of Fifty-Six Websites. J Dent Educ 68: 633-643 [Abstract] [Full text]  
  • Theodosiou, L., Green, J. (2003). Emerging challenges in using health information from the internet. Adv. Psychiatr. Treat. 9: 387-396 [Abstract] [Full text]  
  • Bath, P. A., Bouchier, H. (2003). Development and Application of a Tool Designed to Evaluate Web Sites Providing Information on Alzheimer's Disease. Journal of Information Science 29: 279-297 [Abstract]  
  • Nettleton, S., Burrows, R. (2003). E-Scaped Medicine? Information, Reflexivity and Health. Critical Social Policy 23: 165-185 [Abstract]  
  • Evans, R, Edwards, A, Elwyn, G (2003). The future for primary care: increased choice for patients. Qual Saf Health Care 12: 83-84 [Full text]  
  • Helft, P. R., Hlubocky, F., Daugherty, C. K. (2003). American Oncologists' Views of Internet Use by Cancer Patients: A Mail Survey of American Society of Clinical Oncology Members. JCO 21: 942-947 [Abstract] [Full text]  
  • Al-Shahi, R, Sadler, M, Rees, G, Bateman, D (2002). The internet. J. Neurol. Neurosurg. Psychiatry 73: 619-628 [Abstract] [Full text]  
  • Purcell, G. P, Wilson, P., Delamothe, T. (2002). The quality of health information on the internet. BMJ 324: 557-558 [Full text]  
  • Wilson, P., Risk, A. (2002). How to find the good and avoid the bad or ugly: a short guide to tools for rating quality of health information on the internet * Commentary: On the way to quality. BMJ 324: 598-602 [Full text]  
  • Bland, J M., Altman, D. G (2002). Statistics Notes: Validating scales and indexes. BMJ 324: 606-607 [Full text]  

Rapid Responses:

Read all Rapid Responses

Evidence-based Medical Practice: A Canadian Website Directory and Instrument to rate quality.
France Légaré , Md, MSc, (F)CCMF, et al.
bmj.com, 9 Mar 2002 [Full text]
Overrated Perception of the Internet by Medical Students of a Developing Country
Pedro Horna, et al.
bmj.com, 12 Mar 2002 [Full text]
tools for scoring miscarraige websites
j christopher r. hardwick, et al.
bmj.com, 18 Mar 2002 [Full text]

Student BMJ

Intimate examinations

Israeli students are refusing to perform intimate examinations on anaesthetised women without their informed consent.

www.student.bmj.com

Listen to the latest BMJ Interview