Information in practice

Towards quality management of medical information on the internet: evaluation, labelling, and filtering of information

Hallmarks for quality of information

Quality on the internet

Assuring quality and relevance of internet information in the real world

Towards quality management of medical information on the internet: evaluation, labelling, and filtering of information

Gunther Eysenbach, resident,  Thomas L Diepgen, consultant in dermatology. 

Unit for Medical Informatics, Epidemiology, and Public Health, Department of Dermatology, University Hospital Erlangen, Hartmannstrasse 14, 91052 Erlangen, Germany

Correspondence to: Dr Eysenbach Gunther.Eysenbach{at}derma.med.uni-erlangen.de

The principal dilemma of the internet is that, while its anarchic nature is desirable for fostering open debate without censorship, this raises questions about the quality of information available, which could inhibit its usefulness. While the internet allows "medical minority interest groups to access information of critical interest to them so that morbidity in these rare conditions can be lessened,"¹ it also gives quacks such as the "cancer healer" Ryke Geerd Hamer a platform (http://www.geocities.com/HotSprings/3374/index.htm).^2-4

Quality is defined as "the totality of characteristics of an entity that bear on its ability to satisfy stated and implied needs."⁵ For quality to be evaluated, these needs have to be defined and translated into a set of quantitatively or qualitatively stated requirements for the characteristics of an entity that reflect the stated and implied needs. So how can we define consumers' "needs" in the case of medical information on the internet?

The quality of medical information is particularly important because misinformation could be a matter of life or death.⁶ Thus, studies investigating the "quality of medical information" on the various internet venueswebsites,⁷ mailing lists and newsgroups, ^{8 9} and in email communication between patients and doctors¹⁰are mostly driven by the concern of possible endangerment for patients by low quality medical information. Thus, quality control measures should aim for the Hippocratic injunction "first, do no harm."

Most papers published so far about the problem of quality of medical internet information focus on assessing reliability, but, as box 1 shows, this should be only one aspect of quality measures aiming for "first, do no harm." Another should be to provide context. Although these two problems are different in nature and different measures may be proposed to solve them, we discuss a common measure that could solve both aspects at the same time: assigning "metadata" to internet information; both evaluative metadata to help consumers assess reliability and descriptive metadata to provide context.

Benchmarks

Ideally, the success of methods of quality control and evaluation would be tested by their impact on morbidity, mortality, and quality of life. Such benchmarks would, however, be extremely difficult to measure.¹² Therefore, measures of process and structure¹³ could be used as more indirect indicators of qualityfor example, reliability, provision of context, qualification of authors, use or acceptance of this information by consumers, etc.

Filtering and selecting information

Table 1 shows different systems for quality control of information on the internet. If quality control at the time of production is not possible or not desirable,¹⁴ it could be decentralised and consist of selecting the products complying to the quality requirements of a consumer. Such selection may consist of downstream filtering (by consumers) and upstream filtering (by an intermediary).

Selection by third parties (upstream filtering)
Today, many reviewed indexes (review services) rate medical websites. ^{15 16} In this "upstream filtering" approach, third parties set quality criteria and also perform the evaluations, usually by means of a few human reviewers. This is one possible form of "distributed" quality management, but it has problems (see box 2).

Filtering by the user (manual downstream filtering)
An approach that circumvents some of the problems of upstream filtering (especially that of the volatility of internet information) is that of third parties communicating selection criteria to users (without any attempt to rate internet information themselves) to help consumers to evaluate ("filter") information "manually" on their own.¹⁷ The huge drawback of this approach is that it does not really help consumers to find high quality information quickly, as they have to check manually each entity (website, email, news article) against the given set of quality criteria.

Filtering by the user supported by software (automatic downstream filtering)
We therefore propose to focus on a third approach, automatic downstream filtering. Here, quality criteria are set up by third parties and translated into a computer readable vocabulary, and the filtering is done, at least partly, by users' software.

Electronic labels

The World Wide Web Consortium has recently developed a set of technical standards called PICS (platform for internet content selection)^18-21 that enable people to distribute electronic descriptions or ratings of digital works across the internet in a computer readable form. PICS was originally developed to support applications for filtering out pornography and other offensive material, to protect children. An information provider that wishes to offer descriptions of its own materials can directly embed labels in electronic documents or other items (such as images)for example, such labels may indicate whether the content is appropriate for particular audiences such as minors, patients, etc.

Perhaps even more important, independent third parties, so called label services, can describe or evaluate materialhuman reviewers or automatic software (see below) rate websites and create electronic labels. An end user's software will automatically check at the label bureau(s) that the user is subscribed to while accessing a website or retrieving any other kind of digital information. The software further interprets the computer readable labels and checks them against the requirements defined by the user. It may then, for example, display a warning if the information is aimed at a different audience or if the website is known to contain misleading health information, etc.

The quality criteria (in PICS terms "rating categories") and their scales are together called rating vocabulary. We have developed a prototype core vocabulary, med-PICS, for possible use with medical information.²² This vocabulary contains descriptive categories such as the intended audience (from "kids" to "highly specialised researcher"), which could be used by authors to provide "context," and evaluative categories such as "source rating" (from "highly trustworthy" to "known to provide wrong or misleading information"), which could be used by third party label services.

The main advantages of automatic downstream filtering would be

The exact quality requirements can be set by the user, not by the rating service alone. The rating service describes the information with values on defined scales in different categories, and the user determines the thresholds. For example, a user could tell the software, "I want only material that is suitable for patients, which relates to the healthcare setting in Britain, and which is rated of at least medium reliability"

The software could automatically check one or more rating services in the background, without the user having explicitly to consult a rating service before or after entering a website or retrieving any other kind of information.

The idea of assigning standardised metadata to medical information on the internet is not new,²³ but the key difference of using an infrastructure such as PICS is that not only can authors include metadata but third parties can also associate metadata to all kinds of information (see table 2). Until now metadata were primarily thought of as descriptive (provided by authors), but in the future metadata could also be evaluative (provided by third parties).

Who should evaluate and how

PICS is merely an infrastructure for distributing metadata, not a method per se to evaluate information. The questions of who should evaluate and how still remain.

Today, most of the rating of medical information is done by organisations, publishers, and sometimes individuals. We think that in the future more people from the medical community should evaluate internet information while they surf the internet. We propose a collaboration of medically qualified internet users, consisting of volunteers who, for example, get a program or browser extension that allows them to rate medical websites in a standard format. These ratings could be transmitted to one or several medical label databases, which could be used by consumers.

If thousands of doctors continuously took part in a global rating project we might be able to keep pace with the dynamics of the internet. With this true "bottom up" approach, one could also easily evaluate the rating instruments in terms of variation among observers. Further, the heterogeneity of the reviewers would take account of the many different perspectives and backgrounds that consumers may have as well.

Beyond peer review: automatic and semiautomatic methods of assessing quality

Traditional peer review has many problems, such as that reviewers are human and can make factually incorrect judgments and that peer reviewing is very time consuming. We therefore propose that more work should be made to explore the potential of computers to determine indirect quality indicators by means of automatic (mathematical) methods. Current research suggests that "web surfing" follows strong mathematical patterns,²⁴ and work in the new discipline of "cybermetrics" has indicated promising methods for measuring the impact of websitesdistinguishing low quality websites from high quality sites by analysis of user behaviour, user pattern, complexity of the website, etc (box 3). Of course, the specificity of such indicators is low (a popular website with many users may still harm with unreliable information), but they are sensitive and, once the methods are established and validated, easy to obtain.

Conclusion and call for action

While suggestions for an agreed formal international standard for medical publications on the internet, enforced by appropriate peer or government organisations,²⁶ are probably not realistic, there should at least be a core standard for labelling health related information. In our proposed collaboration for critical appraisal of medical information on the internet,²² organisations, associations, societies, institutions, and individuals interested in reviewing, assessing, and compiling medical information will be invited to join the discussion.

The interneta decentralised medium by naturenot only allows access to information distributed on various computers but also allows a distributed management of quality with decentralised quality control and evaluation. Filtering techniques and infrastructures such as PICS may help to overcome the present oligarchic approach of a few review services attempting to rate all the information of the internet towards a truly distributed, democratic, collaborative rating.

Acknowledgments

Funding: Partly supported by a grant of the German Research Net Association (DFN-Verein), Berlin, and the German Research Ministry (BMBF), Bonn, grant No TK 598-VA/I3.

Conflict of interest: None.

References

1. Dearlove OR, Sharples A, Stone C. Internet is useful for information on rare conditions [letter]. BMJ 1997; 315: 491[Free Full Text].
2. De Bousingen DD. Austrian cancer patient's parents sentenced [news]. Lancet 1996; 348: 1440[Medline].
3. Nigel G. German "quack healer" arrested [news]. Lancet 1997; 349: 1679.
4. De Bousingen DD. German "quack healer" sentenced [news]. Lancet 1997; 350: 874.
5. International Organization for Standardization, Technical Committee ISO/TC 176. ISO 8402: Quality management and quality assuranceVocabulary. 2nd ed. Geneva: International Organization for Standardization , 1994(1994-04-01.)
6. Weisbord SD, Soule JB, Kimmel PL. Brief report: poison on lineacute renal failure caused by oil of wormwood purchased through the internet. N Engl J Med 1997; 337: 825[Free Full Text].
7. Impicciatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the world wide web: systematic survey of advice on managing fever in children at home. BMJ 1997; 314: 1875-1881[Abstract/Free Full Text].
8. Culver JD, Gerr F, Frumkin H. Medical information on the internet: a study of an electronic bulletin board. J Gen Intern Med 1997; 12: 466-470[Medline].
9. Hernandez-Borges AA, Pareras LG, Jimenez A. Comparative analysis of pediatric mailing lists on the internet. Pediatrics 1997; 100: E8.
10. Eysenbach G, Diepgen TL. Responses to unsolicited patient e-mail requests for medical advice on the world wide web. JAMA 1998; 280: 1333-1335[Abstract/Free Full Text].
11. Coiera E. The internet's challenge to health care provisiona free market in information will conflict with a controlled market in health care [editorial]. BMJ 1996; 312: 3-4[Free Full Text].
12. Wyatt JC. Commentary: measuring quality and impact of the world wide web. BMJ 1997; 314: 1879-1881[Free Full Text].
13. Donabedian A. Explorations in quality assessment and monitoring. , Vol 1. The definition of quality and approaches to its assessment Ann Arbor, MI: Health Administration Press , 1980.
14. Silberg M, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the internet. Caveat lector et vieworlet the reader and viewer beware. JAMA 1997; 277: 1244-1245[Medline].
15. Jadad AR, Gagliardi A. Rating health information on the internetnavigating to knowledge or to Babel? JAMA 1998; 279: 611-614[Abstract/Free Full Text].
16. McNab A, Anagnostelis B, Cooke A. Never mind the quality, check the badge-width! Ariadne 1997 May 18;9. (http://www.ariadne.ac.uk/issue9/quality-ratings.)
17. Ambre J, Guard R, Perveiler FM, Renner J, Rippen H. White paper: criteria for assessing the quality of health information on the internet (working draft, edited 14 Oct 1997). http://www.mitretek.org/hiti/showcase/documents/criteria.html [6 Apr 1998].
18. Krauskopf T, Miller J, Resnick P, Treese GW. Label syntax and communi-cation protocols. World Wide Web J 1996; 1(4): 45-69 [28 Aug 1997].)
19. Miller J, Resnick P, Singer D. Rating services and rating systems (and their machine-readable descriptions). World Wide Web J 1996; 1(4): 23-43 [28 Aug 1997].)
20. Resnick P, Miller J. PICS: internet access controls without censorship. Commun ACM 1996; 39(10): 87-93. htm [28 Aug 1997].)
21. Resnick P. Filtering information on the internet. Sci Am 1997; 3: 106-108.
22. Collaboration for Critical Appraisal of Medical Information on the Internet. med-PICS electronic rating vocabulary for medical websites. http://medpics.org [28 Aug 1997].
23. Appleyard RJ, Malet G. A proposal for using metadata encoding techniques for health care information indexing on the WWW. In: American Medical Informatics Association annual fall symposium, Nashville, 25-27 Oct 1997. In: Philadelphia: Hanley and Belfus , 1997.
24. Huberman BA, Pirolli PLT, Pitkow JE, Lukose RM. Strong regularities in world wide web surfing. Science 1998; 280: 95-97[Abstract/Free Full Text].
25. Garfield E. Significant journals of science. Nature 1976; 264: 609-615[Medline].
26. Malet G. Medical matrix code of conduct beta version 1.0 http://206.1.96.60/forums/Thread.cfm?CFApp=3&Thread_ID=33 &mc=2 [28 Aug 1997].

(Accepted 16 July 1998)

Hallmarks for quality of information

J A Muir Gray, director of research and development. 

NHS Executive Anglia and Oxford, Department of Health Institute of Health Sciences, Oxford OX3 7LF

graym{at}rdd-phru.cam.ac.uk

The Goldsmiths' Company was founded in London in 1327 and has flourished for over 650 years. It never traded gold but specialised in the assay of gold and other precious metals. The Goldsmiths' Company has flourished because it has been an independent assay service, measuring the quality of gold and stamping the gold with a hallmark to indicate to the public the purity of the metal with an explicit system of measurement (the word "carat" derives from the Arabic for the carob bean, for the beans of the carob are of uniform size and can be used as standard weights).

Knowledge hallmarks are needed to perform the function of gold hallmarks, and the Cochrane logo has already become a knowledge hallmark, clearly defining the quality of knowledge because readers can look at the Cochrane Collaboration Handbook and see the methods used to produce and appraise the Cochrane Reviews. Journal titles have been another hallmark, but the dependability and credibility of that hallmark is fading as doubts increase about the rigour of the assay method called peer review and evidence shows that even in prestigious journals the assay procedure is flawed and unreliable. Worryingly, all the flaws in the assay procedure seem to overemphasise the strength of the positive effect of new interventions and treatments, with a significant increase in the positive effect of the treatment resulting from poor trial design (table 1) and biased reporting (table 2).

This is a problem in the paper world and will be even more of a problem in the electronic world, in part because electronic journals are so easy to create. Every time information on the world wide web has been critically reviewed or assayed, the quality has been shown to be very variable. Even more worrying, it is hard, and sometimes impossible, to assess the quality of a website because the necessary evidence is not present.

When the printing press was invented, there was concern that the printed word would give undue credibility to an idea or proposition. The same applied to the world wide web when it started, although people now have a healthier scepticism for anything on the web because of the rapid growth of electronic junk. However, the web is an important means of communication, and will become increasingly important when it becomes available on digital television. Already tools have been developed to monitor the quality of healthcare information: DISCERN and the National Centre for Information Quality are examples of initiatives taken to help the public appraise the quality of information provided to them. What is needed, however, is a common standard based on the intellectual equivalent of carob beans, with an Honourable Company of Healthcare Knowledgesmiths to run the assay procedure in an independent and disinterested way so that people can not only distinguish gold from a base metal but also know whether they are reading 24 carat or 18 carat knowledge.

References

1. Schultz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 408-412[Abstract].
2. Gray JAM. Evidence-based healthcare. London: Churchill Livingstone , 1997.

Quality on the internet

Maurizio Bonati, head,  Piero Impicciatore, senior research fellow,  Chiara Pandolfini, research fellow. 

Laboratory for Mother and Child Health, Istituto di Ricerche Farmacologiche "Mario Negri," Via Eritrea 62, 20157 Milan, Italy

Correspondence to: Dr Bonati mother_child{at}irfmn.mnegri.it

Interest in searching the world wide web for health related information continues to increase, increasing the need for internet resources to be accountable to doctors and the public.¹ Function, structure, and content of a website are the main aspects used to evaluate material on the internet.² Although we have not yet developed reliable methods for evaluating the effects (the impact) of such material on clinical practice or on a user's behaviour, improved technology today allows for the control of function and structure of a website.

Eysenbach and Diepgen propose the use of a promising automatic "downstream filtering" system of metadata based on PICS technology. This uses a rating vocabulary that contains descriptive and evaluative categories based on rating instruments already available for evaluating health information on the internet. The authors suggest that assessing quality of information depends not only on evaluating its reliability but also on the provision of context; a valid idea in that it resembles the traditional system of submitting and publishing scientific articles. Thus, providing descriptive tags (metadata) for context and contentlike supplying keywords for articles submitted for publicationwould allow more accurate searches by web browsers.

The problem lies in assigning tags for reliability of information. Guidelines for every aspect of health care do not exist, so each "rater" in the authors' proposed collaboration for critical appraisal of medical information on the internet would assign his or her own values. The benefits of having many raters need to be weighed against the possibility of having unqualified or uninformed medical workers (and lay people) judge web information incorrectly. Who would then check the raters? It has, after all, been found that doctors are also sources of incorrect, outdated information on the internet.³

Thus far, more attention has been paid to presentation and reliability than to the accuracy of the content material.⁴ To determine the accuracy of medical information on the internet we need to compare it with the best evidence.² The evidence based methodology and the Cochrane Collaboration are two useful examples of critical appraisal that should also characterise future evaluations of websites. In the meantime, interaction and feedback may be markers of high quality for websites: allowing a user to submit comments or questions demonstrates a serious intention by the authors to both improve the information supplied by them and to become respectable sources of health information in the long run.

This is a just a starting point for the demystification of medicine and the development of real partnerships between all parties concerned. We must find ways of producing, validating, and diffusing appropriate information in a manner that involves users (consumers) in order to guarantee a non-authoritarian practice, access for all to healthcare information, and high quality information on the internet.

References

1. Morris TA, Guard RJ, Marine SA, Schick L, Haag D, Tsipis G, et al. Approaching equity in consumer health information delivery: NetWellness. J Am Med Inf Assoc 1997; 4: 6-13[Abstract/Free Full Text].
2. Wyatt JC. Commentary: measuring quality and impact of the world wide web. BMJ 1997; 314: 1879-1881.
3. Impicciatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the world wide web: systematic survey of advice on managing fever in children at home. BMJ 1997; 314: 1875-1881.
4. Jadad AR, Gagliardi A. Rating health information on the internet. Navigating to knowledge or to Babel? JAMA 1998; 279: 611-614.

Assuring quality and relevance of internet information in the real world

Subbiah Arunachalam, distinguished fellow. 

M S Swaminathan Research Foundation, Taramani Third Cross Street, Chennai 600 113, India

subbiah_a{at}hotmail.com

Interest in how new information technology can be used to improve health is growing steadily. Telemedicine is making it possible to erase geographical constraints on the provision of health care. However, the information revolution is not a worldwide phenomenon: in India today there are fewer than two main telephone lines per 100 people. Even in Western countries such as the United States there is a wide disparity in terms of access to telephones and computers between poor communitiesinner city populations, blacks, and Hispanicsand the suburban elite.

Access to technology is only a part of the problem. There are three aspects to provision of information: collection, distribution and dissemination, and authentication and quality control. While the internet is good at the first two, the information it provides is not thought to be very dependable or reliable. Eysenbach and Diepgen address this problem with regard to medical information and suggest "distributed quality management" as a possible solution. They argue their casethat questions of both relevance and reliability can be tackled by a common measurevery well. In particular, their proposal that both "top down" and "bottom up" approaches involving peer review by a large body of people should be used is attractive and could be cost effective.

There are good examples of achieving quality assurance through a combination of centralised and decentralised approaches in other specialties. The United Nations Environment Programme has the maESTro (Managing Environmentally Sound Technologies) program, which operates from Japan and which verifies with the developer of the technology as well as cross checks with databases (http://www.unep.or.jp/ietc/ESTdir/maestro/introduction.html). In physics the e-Print Archive, based in Los Alamos, works well. Usually, if someone wants to comment on a preprint, he or she directs it to the author, but some do forward their comments to the archive, thus making it available to the worldwide audience.

Unlike in physics and technological information services, in medicine a whole range of people, and not only experts, take part in the information exchange, both inputting and searching. Well known sites such as those of the BMJ, JAMA, and Human Genome News (http://www.ornl.gov/TechResources/Human_Genome/publicat/hgn/vgn3/01eyes.html) are dependable, but what about all the material in usenet groups, listservs, and email messages? In this respect medicine is closer to astrology than to the hard scienceshence the need for assuring quality. We should encourage doctors and biomedical researchers, as well as institutions, to comment on what they see on the internet. Also, agencies such as Magellan and Starting Point (web search engines that also evaluate websites) perform the function of third party evaluators. Ultimately, the reliability of the meta-analysis approach (gaining new insights by amalgamating existing data from different sources) would depend on the weights we give to differ-ent constituents in the distributed, democratic, and collaborative process of rating suggested by Eysenbach and Diepgen. Another problem, not just with medical information but with any information, is the cost of standardisation of vocabulary, evaluation procedures, etc. Who will pay?

Certain new developments in searching the world wide web, such as the "hyperlink induced topic search" developed by Jon Kleinberg of Cornell University and being evaluated by IBM and Digital (now Compaq) for implementation, can help to reduce the time taken to find relevant medical information in an internet search (see http://www.almaden.ibm.com/cs/k53/clever.html.). This is similar to the citation links in journal literature that form the basis of the Science Citation Index. But it is still unclear whether the system for the internet will be as powerful as the citation indexes in clustering related material through cognitive links.

Finally, we live in the real world, and there can be no ideal solution to our problems. Every time we find a way to overcome a problem, those that create the problem do things to make our solutions inadequate. But scepticism should not hold us back from looking for ways to make the internet the ultimate source of easily accessible and reliable information.