Information in practice

Verifying quality and safety in health informatics services

Michael Rigby, senior lecturer ^a, Jari Forsström, director ^b, Ruth Roberts, lecturer ^c, Jeremy Wyatt, director ^d,  for the TEAC-Health Partners.

^a Centre for Health Planning and Management, Keele University, Keele ST5 5BG, ^b Medical Informatics Research Centre in Turku, University of Turku, FIN-20520 Turku, Finland, ^c School of Postgraduate Studies in Medical and Health Care, University of Wales Swansea, Swansea SA2 8PP, ^d Knowledge Management Unit, School of Public Policy, University College London, London WC1H 9QU

Correspondence to: M Rigby m.j.rigby{at}keele.ac.uk

Information and its handling and transmission form an essential part of health care and are reflected in professional standards. Automated information systems in health carehealth informatics serviceswill improve these functions and bring new opportunities through the harnessing of modern information and communications technologies. Thus, computer support is now essential in many parts of medicine, the US Institute of Medicine has long espoused the value of computerised patient records,¹ and many countries have developed strategies on this topic, and there are countless health related internet sites.

However, as new information and communication technologies in health bring new opportunities, they also bring new risks. Emphasis has rightly been placed on ensuring appropriate levels of confidentiality in electronic information systemsto the point that the highly exacting requirements being demanded by independent commentators and professional bodies² are difficult to satisfy without jeopardising the functioning of core services ^{3 4} or the interests of the most vulnerable groups.⁵ In contrast, much less thought has been given so far to ensuring the appropriateness of the design and integrity of functioning of health informatics services.

	Importance of quality assurance of health informatics systems

Top Importance of quality assurance... The TEAC-Health project Classification of health... A taxonomy of risk... Quantification of the problem TEAC-Health recommendations for... TEAC-Health recommendations for... TEAC-Health recommendations for... Conclusions References

If informatics systems are increasingly essential in the delivery of health care then their integrity and quality must be of equal importance, but this has been scarcely recognised to date. In 1963 the then UK secretary of state for health stated to the House of Commons: "The House and the public suddenly woke up to the fact that any . . . manufacturer could market any product, however inadequately tested, however dangerous, without having to satisfy any independent body as to its efficacy and safety and the public was almost uniquely unprotected in this respect."⁶ That statement related to drugs, being triggered by the thalidomide disaster, and the situation was changed rapidly. However, the same situation applies today with regard to electronic health informatics products and services, which are now the most important unregulated healthcare resourcein sharp contrast to drugs, medical devices, and licensed health professionals.

When errors and failures have occurred it has generally been in the interests of suppliers, provider organisations, and clinicians to quietly rectify or remove the flawed systems rather than draw attention to them. This, however, allows for unidentified and thus unquantified errors to be dispersed, with potential risk to patient health. Box 1 gives published examples of such health threatening errors in computer software. In a modern consumerist environment, however, this situation is unacceptable, as shown by the public furore over the software that miscalculated the risk of Down's syndrome in pregnancies.¹⁰

	The TEAC-Health project

Top Importance of quality assurance... The TEAC-Health project Classification of health... A taxonomy of risk... Quantification of the problem TEAC-Health recommendations for... TEAC-Health recommendations for... TEAC-Health recommendations for... Conclusions References

Recently, a European projecttowards European accreditation and certification of telematics services in health (TEAC-Health)was conducted to investigate the issues, and we report its core findings here. The findings outlined in the project report¹¹ have recently been formally accepted by the European Commission, which intends to examine in detail the steps required for their implementation (Jean-Claude Healy, head of health applications unit, Information Society Directorate-General, personal communication, 2000).

The project arose from an expert conference at Turku University in 1997 organised by JF, which resulted in several published articles.^12-14 The work of the project was undertaken by representatives of five European countries; details of the membership and working reports can be found on the Multimedica website.¹⁵

	Classification of health informatics services

Top Importance of quality assurance... The TEAC-Health project Classification of health... A taxonomy of risk... Quantification of the problem TEAC-Health recommendations for... TEAC-Health recommendations for... TEAC-Health recommendations for... Conclusions References

For the project, we classified health informatics services into three categoriessoftware and related services, telemedicine, and internet sites. Although many services combine more than one of these elements, the quality assurance and regulatory components for each need to be considered separately as the issues are quite distinct. We also felt it inappropriate to consider processes of quality assurance and verification solely in the health sector and therefore looked at commercial approaches such as regulation in the financial sector and at other areas of public risk such as air traffic control and food safety.

In the health sector, precedents have been set in the regulation of drugs and medical devices, but neither of these is directly applicable to health informatics services. Safety control of new drugs now depends largely on controlled trials, which are neither feasible nor affordable as a mandatory control for clinical software or internet sites. Regulation of medical devices has several similarities, but key differences are the much wider range of user proficiency and circumstances of use of informatics systems compared with medical devices and the difficulties of ensuring structured user training and education.

	A taxonomy of risk assessment

Top Importance of quality assurance... The TEAC-Health project Classification of health... A taxonomy of risk... Quantification of the problem TEAC-Health recommendations for... TEAC-Health recommendations for... TEAC-Health recommendations for... Conclusions References

We next considered how best to categorise risk in health informatics services, as it is only by identifying risk that appropriate control methods can be identified. For medical devices, the regulations are clear and helpful: they require that a device's manufacturer or supplier identifies the risk level as determined by the type of product and how life critical are the circumstances of its use.¹⁶ We concluded that risk in health informatics services depends on a combination of type of user, circumstances of use, type of use, and nature of the system. For example, a failure in an automated appointments system can have serious consequences by passing undetected, whereas an experienced clinician may filter out spurious results from a diagnostic support tool used merely as an aide memoire. The table shows the different levels of risk associated with different health informatics services.

	Quantification of the problem

Top Importance of quality assurance... The TEAC-Health project Classification of health... A taxonomy of risk... Quantification of the problem TEAC-Health recommendations for... TEAC-Health recommendations for... TEAC-Health recommendations for... Conclusions References

We sought to identify and quantify the risks attributable to informatics services, and the degree of concern they produced. A comprehensive literature search and a small targeted survey of European opinion leaders from health and consumer domains showed that the problem was, if anything, greater than anticipated.¹⁵

Clinical software
Many of the problems identified when using clinical software are resolved between supplier and user on condition that there is no publicity, while the problems that are not identified cannot, by definition, be reported. Thus the literature will substantially underestimate these problems, but some errors have been reported (see box 1), as has the adverse outcome of software upgrades producing erroneous printed interpretations of previously recorded diagnostic data.¹⁷

Telemedicine
Less has been published about the risks of telemedicine services because of their comparative newness. However, we identified concerns about authenticity and risks in telemedicine services, including email consultations, other than those within a single provider organisation or on a closed, point to point basis.¹⁵ There are indications that a quarter of those offering telemedicine consultations directly to the general public do not hold the qualifications they claim (S Schanz, personal communication, 2000), and others may be offering advice beyond their qualifications. Studies have shown there is wide variation in the quality of advice provided, and, although guidance may generally be sound, the occurrence of so many outliers is an unacceptable and avoidable risk. ^{18 19}

Internet sites
Services on the world wide web are the most obvious risk, as anyone can publish any information they like. Much of this information is valuable and the internet allows freedom of expression for patient support groups and leaders in alternative therapies, but studies have shown that both misleading and life threatening advice is readily available. ^{20 21} A figure of 1400 "suspicious" websites was reported by the coordinator of a study for the G8 group of countries, with a 21% increase in that number annually,²² and a recent US study found errors and contradictions even within sites.²³ Yet, by its very nature, the internet cannot be controlled or censored.

Project survey
Our survey of opinion leaders, for which we used a "snowball sample" method, yielded 54 respondents, of whom 36 (67%) indicated that they had experienced one or more problems with health telematics services. Of the 74 problems reported, 10 adversely affected patient safety, four adversely affected optimum treatment of a patient, and 31 adversely affected the health professional's duty of care to a patient. Of all the respondents, 19 were "very concerned" about the current lack of quality assurance of telematics services and a further 22 had some concerns, giving a total of 41 (76%) "concerned."

	TEAC-Health recommendations for clinical software

Top Importance of quality assurance... The TEAC-Health project Classification of health... A taxonomy of risk... Quantification of the problem TEAC-Health recommendations for... TEAC-Health recommendations for... TEAC-Health recommendations for... Conclusions References

In view of the need to avoid identified risks to the public, and the professional opinion in favour of some form of regulation, we concluded that specially crafted regulation was needed based on existing European experience with product control and monitoring health risks. The components suggested are as follows.

CE marking
Applying this publicly understood and reliable mark on approved goods is a well established process in Europe based on clear regulation²⁴ and with variants for medical devices.¹⁶ However, further research is needed on the specific criteria to accommodate clinical software. This will require a "notified body" to have overall responsibility and to identify and monitor essential requirements for these products and services. As concurrent verification of design and quality is far more effective than retrospective testing, the necessary identification of control measures for production and quality assurance will itself yield invaluable standards for clinical software developers.

Labelling
A legally underpinned requirement for accurate and detailed labelling is a key element of our proposed solution, as this will enable purchasing organisations and clinical users to know much more about the software product. Identification of named responsible individuals will also substantially increase the commitment to ensure quality of design and manufacture. The exact requirements will need further discussion and definition, but box 2 shows a suggested list.

"Hotline" for postmarketing surveillance
An essential part of CE marking is postmarketing surveillance, in particular the requirement that the supplier provides a "hotline" telephone number to which any problem or concern can be reported. It is also a statutory requirement of CE marking that all serious incidents are reported by the supplier to a "competent authority," and this process is liable to unannounced audit on site.

National hotlines and monitoring organisations
Based broadly on existing models for drug products and medical devices, national hotlines and monitoring organisations are necessary for clinical software to ensure that problems such as adverse interactions between different products (see box 1) can be identified speedily. They are of proved benefit for other clinical products and already apply to health software in Sweden.

In house software and informatics services
Software and services developed by particular healthcare organisations for their own use cannot readily be subjected to compulsory CE marking as they are not marketed products. However, our proposed regulation would bring two safeguards. Firstly, the identification of professional standards would form a yardstick for identifying reasonable practice and duty of care should there be a formal complaint or litigation. Secondly, in house products could be submitted voluntarily to the verification process.

	TEAC-Health recommendations for telemedicine

Top Importance of quality assurance... The TEAC-Health project Classification of health... A taxonomy of risk... Quantification of the problem TEAC-Health recommendations for... TEAC-Health recommendations for... TEAC-Health recommendations for... Conclusions References

Telemedicine presents an entirely different situation because telecommunications based services that cross legislative boundaries are almost free of regulation. Thus, providers of healthcare services could escape regulation, particularly when moving to the internet. Since this leaves patients at risk, some control mechanisms are needed. In principle, legislation should be independent of the communication medium usednamely, the same ethical principles and liabilities should apply to telemedicine as to conventional patient care. Because telemedicine services can readily cross international boundaries, international coordination or coregulation is needed in Europe and beyond. Similarly, in countries such as the United States regulation is at the state level, leading to complex and unwieldy situations that hamper legitimate national providers and thus also patients.

Key elements of regulating telemedicine services should include international agreement as to whether such services are delivered under the law of the supplier or that of the consumer. The European Permanent Committee of Physicians (EPCP) now favours accepting European law that it is the supplier's legal system that applies (Ä Markku, chairman, EPCP, personal communication, 2000). Secondly, labelling (as above) with legal sanctions should be required, linked to a code of conduct, which needs to be developed. Box 3 shows proposed key elements.

Global regulation
A global regulatory framework is also important. There are clear and effective global conventions and supervisory organisations for both civil aviation and food standards, both of which operate on an evidence based principle, obtaining and interpreting emergent scientific evidence in order to formulate new standards that then become the basis for universally agreed international regulation. Delivery of telemedicine services internationally puts individual patients at risk of injury or death through incompetent or malicious unregulated providers, but, because the transactions are individual and confidential, adverse outcomes are not as conspicuous as in domains such as civil aviation. The same situation applied to pharmaceutical products until regulation.⁶ The global risk to personal health continues unabated in the absence of international agreement on regulation, liability, and control. We consider international telemedicine to deserve at least the same level of regulation as the civil aviation and food sectors. This could also aid the development of national frameworks, especially in countries with largely independent states or provinces.

	TEAC-Health recommendations for internet sites

Top Importance of quality assurance... The TEAC-Health project Classification of health... A taxonomy of risk... Quantification of the problem TEAC-Health recommendations for... TEAC-Health recommendations for... TEAC-Health recommendations for... Conclusions References

We believe that the cost of developing a system solely to verify the quality of health internet sites would be high and that it would be impractical. The Health on the Net Foundation (HON) has for some time been promoting a voluntary code of conduct, and there have been several overlapping initiatives in the United States (see box 4), but their main drawback is that there is no external verification and so the system is open to abuse and, indeed, offers false security.

However, the need for independently verified sites is common to many other internet activities, including retailing.²⁵ As with CE marking and other recognised quality standards, the power of effective regulation depends on the universality of use leading to public recognition. We studied earlier attempts to identify high quality sites to the public, the best known being filtering mechanisms and rating systems.¹¹ Both have drawbacks.

Most filtering excludes inappropriate items but also excludes many relevant sites, as it is difficult to develop a 100% specific yet sensitive filter that does not filter out required material. For example, a filter designed to protect against pornography will exclude sites with the word "breast," but it will also filter out important medical sites. Such "heuristic" filtering depends on finding and interpreting key words. The alternative, "filtering in," requires the site to undertake self rating honestly and accurately.

Rating systems depend on third parties such as informed users to provide a rating and score for each individual site, but this raises questions of ensuring objectivity, impartiality, and common clinical and cultural values to the extent that there are now proposals for rating the raters. Moreover, this leaves most sites unrated. Clearly, these methods are not feasible to aid general public users, nor indeed most health professional users unfamiliar with the intricacies of the internet. Box 5 summarises the issues.

The EuroSeal proposal
We have therefore proposed development of a new European system and standard, entitled the EuroSeal. ^{12 15} This would be a seal supplied to a website by an accredited agency (the approach fundamental to CE marking). Once attached to the site, its integrity would be verified by secure single socket layer or similar secure software, as currently happens with secure trading sites. The seal would be provided at two levels, the higher of which would require independent onsite verification (for a higher fee). The verification processes would be open and transparentby clicking on the EuroSeal symbol, visitors to the site would see details of the site inspections, drawn in real time from the records of the accrediting body (as applies with current secure links for web commerce), as well as the code(s) of conduct to which the site adhered.

Codes of conduct
These are an important element of the EuroSeal approach, as they would form the basis on which the third party assessed a site provider's claims and decided whether to award the EuroSeal. Each health professional body would be able to devise its own codes of conduct and standards, and viewers would know against which code the EuroSeal had been applied. This approach would also allow special interest groupssuch as ethnic groups, those with particular religious beliefs, and advocates of alternative medicineto devise their own codes of conduct. Patient support groups could also devise codes of conduct, provided they met a prescribed framework and standard for codes.

	Conclusions

Top Importance of quality assurance... The TEAC-Health project Classification of health... A taxonomy of risk... Quantification of the problem TEAC-Health recommendations for... TEAC-Health recommendations for... TEAC-Health recommendations for... Conclusions References

Health informatics systems are invaluable to aid health care. Moreover, they bring intrinsic advantages, such as electronic records being more accessible than paper ones and, if properly protected and encrypted, being more secure from damage or prying. However, this is no excuse not to address current known and avoidable risks.

The TEAC-Health project has clearly shown that public safety and professional integrity are threatened by the lack of regulation of health informatics services. These risks will increase rapidly as health informatics services expand and as telecommunications and globalisation radically change attitudes to and delivery of health care. ^{26 27} Initiatives to date have been based on restricted research, lacked consideration of overall feasibility and other issues, or depend on the (usually unpublished) integrity and values of a secondary service provider. The strategic proposals we describe, which have now been welcomed by the European Commission, form an evidence based solution.

	Footnotes

   Competing interests: JW has a small part of the equity of Medix, an internet service provider for doctors, and receives research and consultancy funding from various commercial sources.

	References

Top Importance of quality assurance... The TEAC-Health project Classification of health... A taxonomy of risk... Quantification of the problem TEAC-Health recommendations for... TEAC-Health recommendations for... TEAC-Health recommendations for... Conclusions References

1.	Dick RS, Steen EB, eds. The computer-based patient recordan essential technology for health care. Washington DC: National Academy Press, 1991.
2.	Anderson RJ. Security in clinical information systems. London: British Medical Association, 1996.
3.	Rigby M. Keeping confidence in confidentiality: linking ethics, efficiency, and opportunity in health care computinga case study. In: Anderson R, ed. Personal medical informationsecurity, engineering, and ethics; personal information workshop, Cambridge, UK, June 21-22, 1996 proceedings. Berlin: Springer-Verlag, 1997:129-150.
4.	Roberts R, Thomas J, Rigby M, Williams J. Practical protection of confidentiality in acute care. In: Anderson R, ed. Personal medical informationsecurity, engineering, and ethics; personal information workshop, Cambridge, UK, June 21-22, 1996 proceedings. Berlin: Springer-Verlag, 1997:67-78.
5.	Rigby M, Hamilton R, Draper R. Towards an ethical protocol in mental health informatics. In: Cesnik B, McCray AT, Scherrer J-R, eds. Medinfo 98 9th world congress in medical informatics, proceedings. Amsterdam: IOS Press, 1998:1223-1227.
6.	House of Commons official report (Hansard). Session 1962-6. , 1963 May 8. London: HMSO, 1963.
7.	Hawking M. Code conversions, data stability, and the futurean agenda for discussion. J Inf Primary Care 1995;June:3-5.
8.	Cavalli P. False-negative results in Down's syndrome screening. Lancet 1996; 347: 965-966[Medline].
9.	Computer error leads to smear recalls failure. Health Serv J 1998; 106: 6.
10.	Wilkinson P. Down's test leaves 150 women in abortion fear. Times, 2000 May 31: 1, 3.
11.	Forsström J, Rigby M, Roberts R, Nilsson S, Wyatt J, Beier B, et al. Towards evaluation and certification of telematics services for health (TEAC-Health)key recommendations. Turku: University of Turku, 1999.
12.	Forsström J. Why certification of medical software would be useful? Int J Med Inf 1997; 47: 143-152[Medline].
13.	Wyatt J. Quantitative evaluation of clinical software, exemplified by decision support systems. Int J Med Inf 1997; 47: 165-173[Medline].
14.	Forsström J, Rigby M. Considerations on the quality of medical software and information services. Int J Med Inf 1999; 56: 1-3, 169-76.
15.	Multimedica. Towards evaluation and certification of healthcare applications in Europe. www.multimedica.com/en/ (accessed 10 Aug 2001)
16.	EU Council. Directive 93/42/EEC concerning medical devices. Brussels: European Commission, 1993.
17.	Hawking M. Organisation of general practice: implications of IM&T in the NHS. In: Anderson R, ed. Personal medical informationsecurity, engineering, and ethics; personal information workshop, Cambridge, UK, June 21-22, 1996 proceedings. Berlin: Springer-Verlag, 1997:56-65.
18.	Eysenbach G, Diepgen TL. Responses to unsolicited patient e-mail requests for medical advice on the world wide web. JAMA 1998; 280: 15, 1333-5.
19.	Sandvik H. Health information and interaction on the internet: a survey of female urinary incontinence. BMJ 1999; 319: 29-32[Abstract/Free Full Text].
20.	Impiccatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the world wide web: systemic survey of advice on managing fever in children at home. BMJ 1997; 314: 1875-1879[Abstract/Free Full Text].
21.	Weisbord SD, Soule JB, Kimmel PL. Poison on lineacute renal failure caused by oil of wormwood purchased through the internet. N Engl J Med 1997; 337: 825-827[Free Full Text].
22.	Rogers R. A global information society for healthrecommendations for international action. Br J Healthcare Computing Information Manage 1999; 16: 28-30.
23.	Berland GK. Health information on the internet: accessibility, quality, and readability in English and Spanish. JAMA 2001; 285: 2612-2621[Abstract/Free Full Text].
24.	Council of the European Communities. Decision of 22 July 1993 concerning the modules for the various phases of the conformity assessment procedures and the rules for the affixing and use of the CE conformity marking, which are intended to be used in the technical harmonization directives. Brussels: European Commission, 1993.
25.	Institute of Chartered Accountants in England and Wales. International chartered accountancy bodies launch webtrusta worldwide web assurance service. www.icaew.co.uk/news/document.asp?WSDOCID=1653 (accessed 15 Aug 2001).
26.	Rigby M. The management and policy challenges of the globalisation effect of informatics and telemedicine. Health Policy 1999; 46: 97-103[CrossRef][Medline].
27.	Rigby M. And into the 21st centurytelecommunications and the global clinic. In: Rigby M, Roberts R, Thick M, eds. Taking health telematics into the 21st century. Oxford: Radcliffe Medical Press, 2000.

(Accepted 19 July 2001)