Editor's Choice | This Week in BMJ | Press releases
BMJ No 7098 Volume 314 Information in practice Saturday 28 June 1997 Commentary: Measuring quality and impact of the world wide webJeremy C Wyatt
The world wide web gives patients and professionals
access to thousands of pages of clinical information, some of which are
assessed by Impicciatore et al above. (1) However,
although the web makes it absurdly easy to disseminate information, by
allowing anonymous authors to conceal commercial or other conflicts of
interest(2) it does not help readers to discriminate between
genuine insight and deliberate invention.(3) Thus, recent
proposals for improving the accountability of medical information
on the internet(2) will enhance its value. Sometimes,
though, checking whether a web site passes the criteria of Silberg et
al for explicit authorship and sponsorship, attribution of sources, and
dating of material(2) is not enough, as I Thus, for many purposes, evaluation of web sites needs to go beyond
mere accountability to assessing the quality of their content,
functions, and likely impact (see table 1) - similar to the
assessment of electronic textbooks, telemedicine, and decision
support systems, (4-6) where the same issues
arise.
Since internet philosophy declares that anyone can set up a web
site(7) there is a risk that, through ignorance or bias, the
content of the site may not be correct even if the original information
sources were reliable. Impicciatore et al showed that parents searching
for information about treating a feverish child could either receive
good advice or be advised to administer aspirin, putting their child at
risk of Reye's syndrome, according to which web site they
visited.(1) These investigators compared the information
available on each site with statements in a reputable textbook, but
such statements often disagree with contemporary systematic reviews of
the literature.(8) Thus, to determine the accuracy of web
material we need to compare it with the best evidence, which usually
means a meta-analysis of the appropriate kind of evidence. For
effectiveness of treatment this is randomised trials,(9) but
for risk factors it is cohort studies, and for diagnostic accuracy it
is blinded comparisons of the test with a standard.(10)
An important advantage of publishing on the internet is that it
allows regular, even hourly, updating,(7) so that patients
and professionals using the world wide web expect material to be more
up to date than paper sources. The easiest way to assess timeliness is
to check the date on web pages,(2) but, since the material
may not have been current even then, independent comparison with the
most up to date facts obtained elsewhere is preferable.
Even if the content is correct and up to date, people must be able to
read and understand it. The web allows information to be communicated
in many ways - as diagrams, animations, linked pages, flashing red
capitals on a blue background, etc - which may not always improve
legibility and comprehension.(11) Asking visitors to a web
site to record their satisfaction with the material is unlikely to
reveal problems with comprehension, as visitors may not realise that
they have misunderstood or may blame themselves. For web sites intended
for the general public, it is useful to decide a minimum reading age
for the material; a word processor's grammar checker can then be used
to assess the text's readability and reading age. This is often
underestimated; for example, the minimum reader age for this paragraph
is 18 years. However, such measures are less revealing than asking
subjects to answer q One major concern of web site developers is how e Since some web sites are complex, a second question is how easily users
can locate relevant material within the site. It is useful to compare
users' ease of navigating through the site with the ease of using a
printout of the material or the paper documents from which the web site
is derived, to judge if the electronic medium makes information easier,
or more difficult,(12) to locate.
A third functional issue is whether the web site is actually used, and
by whom. Most "server" software for web sites logs each access to
each page together with the abbreviated internet address of the
requesting computer. However, such records of use must be interpreted
carefully: accesses to a page may be accidental, casual browsing by
"info-tourists," or by users en route to another page. Since most
server logs do not distinguish repeated visits to a page by the same
individual, visits to a page cannot be equated with visitors. To
collect more information, users can be asked to fill in web forms, but,
as with paper questionnaires, most usually fail to do this, casting
serious doubt on the generality of the data.(13) Even if
data on use are genuine, comparison of rates of use between different
sites needs to be simultaneous rather than historical, given the
exponential growth in the use of the internet. For those investing resources in a web site, a key question is its
likely impact on clinical processes and patient outcomes and its cost
effectiveness compared with other methods for delivering the same
information.(5) Tentative answers to this question can be
obtained by studying the impact of the site on the knowledge of sample
users in laboratory settings, but its real impact on clinical practice
can be studied only in the field. Randomised trials comparing the
effects of providing the same information in two different ways raise
problems familiar to evaluators of other kinds of information
resource,(4-6) such as contamination of the management of
patients in one arm of the trial by the management of patients in the
other arm, and Hawthorne effects. There do not seem to be any published
trials of the effects of the world wide web on clinical practice, but
such assessments are clearly essential to justify large scale
expenditure on computer networking and web sites and to define adverse
effects. There are two key issues common to many evaluation studies:
choosing appropriate subjects and making reliable, valid
measurements.
Choosing appropriate subjects
Making reliable, valid measurements
Although surfing the web provides an excellent method for patients
and professionals to access clinical knowledge,(2, 7) unless
we evaluate the quality of clinical sites and their effects on users,
we risk drowning in a sea of poor quality information. Improved
technology is not the answer to making better use of this enticing
resource. We need to be clearer about the web's clinical role and the
evaluation problems that it raises - how to recruit suitable subjects,
develop valid and reliable methods of measurement, and carry out many
more rigorous evaluations.
ICRF Centre for Statistics in
Medicine, References
1 Impicciatore P, Pandolfini C, Casella N, Bonati M.
Reliability of health information for the public on the world wide web:
systematic survey of advice on managing fever in children at home.
BMJ 1997;314:1875-9.
2 Silberg W M, Lundberg G D, Musacchio R A. Assessing, controlling
and assuring the quality of medical information on the internet.
JAMA 1997;277:1244-5.
3 Bower H. Internet sees growth of unverified health claims.
BMJ 1996;313:381.
4 Wyatt J C, Spiegelhalter D. Evaluating medical expert systems:
what to test and how? Med Inf 1990;15:205-17.
5 Wyatt J C. Commentary: Telemedicine trials - clinical pull or
technology push? BMJ 1996;313:1380-1.
6 Friedman C, Wyatt J. Evaluation methods in medical
informatics. New York: Springer-Verlag, 1997.
7 Pallen M. Guide to the internet: the world wide web.
BMJ 1995;311:1552-6.
8 Antman E M, Lau J, Kupelnick B, Mosteller F, Chalmers T C. A
comparison of results of meta-analyses of randomized control trials and
recommendations of clinical experts. JAMA
1992;268:240-8.
9 Sackett D L. Rules of evidence and clinical recommendations for
use of antithrombotic agents. Arch Intern Med
1986;146:464-5.
10 Jaeschke R, Guyatt G H, Sackett D L. Users' guides to the
medical literature. III: How to use an article about a diagnostic test.
A: Are the results of the study valid? JAMA
1994;271:389-91.
11 Tufte E R. Visual explanations. Cheshire, CT:
Graphics Press, 1997.
12 Jones R G, Nicholls J, Howes M. Medical information.
Lancet 1992;339:248.
13 Oppenheim A N. Questionnaire design, interviewing &
attitude measurement. London: Pinter, 1991.
|
||||||||||||||||||||||||||||||||||||||