“User-testing” as a method for testing the fitness-for-purpose of written medicine information

Abstract

Objective

Written medicine information is essential to support spoken information from pharmacists, but must be fit-for-purpose. This study applied “user testing” to the booklet supplied to UK patients taking anticoagulant medicines.

Methods

“User testing” uses mixed-methods, applied iteratively, to assess document performance – can people find and understand key points of information through a questionnaire and short semi-structured interview. The booklet was tested in 3 rounds of 10 people. After each round it was revised according to participants’ responses, and re-tested.

Results

The first round questionnaire identified problems with 6/18 information points (booklet purpose; other information; what affected daily doses; effect of ibuprofen; tablet colour; drinking alcohol); interviews raised further issues. The booklet was revised and, in the second testing round, one problem identified (changing doses of other medicines); the interviews raised fewer issues. After further re-wording and re-design, a third round showed all questions found and understood by at least 8/10 participants.

Conclusion

User testing assesses whether people can find and understand key information and can be applied using small numbers of participants. Application to medicine information can markedly improve performance.

Practice implications

Information producers should consider user testing to ensure documents are ‘fit for purpose’ in informing patients.

Introduction

Written medicine information is an important support to spoken information from pharmacists and other health professionals. Patients want such written information to help them decide whether the medicine is right for them, and to help them take the medicine safely and effectively [1]. However for written information to be effective, it must be understandable.

Previously, testing written health information has had a content-based approach. This included applying a checklist to ensure that the appropriate information is present. Another content-based approach is the use of readability formulae such as the Flesch, SMOG or FOG indexes. These formulae calculate a readability score relating to the length of the words and the length of the sentences in the text [2]. Unfortunately they do not actually determine whether potential users of the information can find and understand the information they need. Indeed readability formulae, in particular, fall down because they only measure two of the many factors which influence readability (word and sentence length). This is demonstrated by the way that a piece of text written backwards would achieve the same score as when written forwards: because the text backwards contains the same words and has the same sentence length [3].

From 1999, comprehensive leaflets for the patient supplied with every medicine pack (and written by the manufacturer according to strict guidelines), have been mandatory across the European Union (EU) [4]. Subsequently, an EU law in 2005 required all such information to be tested “with target patient groups to ensure [leaflets are] legible, clear and easy to use”. This required a performance-based method, which measures how a leaflet performs – not just its content. It addresses whether people can find and understand the information they need in the leaflet. Without such a successful and documented test, authorisation to market a new drug will not now be granted in the EU [5]. Since 2005, thousands of medicine leaflets have been through a type of performance-based testing called “user-testing”, which was developed in Australia in the 1990s by Sless and colleagues [6]. This includes all the leaflets accompanying anti-coagulant medicines.

User-testing is based on individual interviews with cohorts of 10 participants. First, a questionnaire is administered to determine whether the individual can find and understand selected key points of information (usually 12–15 but dependent on document length and purpose). Second, a short semi-structured interview is conducted, to explore participants’ views. Participants are those from the target group for the medicine, rather than people actually taking the medicine (as prior use and knowledge would not ensure a fair test). Importantly, user-testing is designed to be iterative, and used in the context of good practice in writing and information design. After each round of testing the findings (both quantitative and qualitative) are analysed, and good practice applied to revise the leaflet to overcome identified problems. Then a further cohort of participants is interviewed and the process repeated until any problems have been satisfactorily resolved [3], [5], [6].

Anticoagulant medicines have a narrow therapeutic window, present a significant risk of potential harm and failure of treatment [7], and are frequently identified as causing preventable harm and admission to hospital [8], [9]. As a consequence they are one of a small number of medicines routinely supplied with written information additional to the pharmacist's label and the manufacturer's leaflet inside the pack. Internationally, such additional information is produced by national agencies in Australia (booklet) [10], the USA (leaflet) [11] and in the UK (known as the ‘Yellow Booklet’) [12].

In the UK, the booklet includes information particularly related to the need for monitoring of the INR (International Normalised Ratio) and associated dosage adjustment, along with information about food and drug interactions, amongst other safety issues. The UK booklet was updated by the British Society for Haematology and the National Patient Safety Agency (NPSA) in 2007 [13]. However, its effectiveness and usability with patients have not been formally tested. This contrasts with the mandatory leaflets supplied in the packs of all anticoagulant medicines in the UK, which have now undergone such user-testing (as described above).

In this paper, the NPSA document about oral anticoagulants is used to demonstrate the application of user-testing for such patient-focused written medicine information. The research questions being asked are, firstly, whether people who are potential users of anticoagulants can find and understand key pieces of information related to safe and effective use. Secondly, if there are problems with the booklet, can applying user-testing iteratively (with its quantitative and qualitative components) inform the development of a revised document which overcomes these problems? This would mean that the booklet would have undergone the same testing process as the patient package leaflets for anti-coagulant medicines.