Papers

Evidence based purchasing: understanding results of clinical trials and systematic reviews

^a Department of Public Health Medicine and Health Policy, Oxfordshire Health, Oxford OX3 9DZ, ^b Department of Social Medicine, University of Bristol, Bristol BS8 2PR

Correspondence to: Dr T Fahey, Department of Social Medicine, University of Bristol, Bristol BS8 2PR.

Abstract

Objective: To assess whether the way in which the results of a randomised controlled trial and a systematic review are presented influences health policy decisions.
Design: A postal questionnaire to all members of a health authority within one regional health authority.
Setting: Anglia and Oxford regional health authorities.
Subjects: 182 executive and non-executive members of 13 health authorities, family health services authorities, or health commissions.
Main outcome measures: The average score from all health authority members in terms of their willingness to fund a mammography programme or cardiac rehabilitation programme according to four different ways of presenting the same results of research evidence--namely, as a relative risk reduction, absolute risk reduction, proportion of event free patients, or as the number of patients needed to be treated to prevent an adverse event.
Results: The willingness to fund either programme was significantly influenced by the way in which data were presented. Results of both programmes when expressed as relative risk reductions produced significantly higher scores when compared with other methods (P<0.05). The difference was more extreme for mammography, for which the outcome condition is rarer.
Conclusions: The method of reporting trial results has a considerable influence on the health policy decisions made by health authority members.

Introduction

The randomised controlled trial is regarded as the gold standard in the assessment of healthcare interventions.¹ Its explanatory power permits qualitative conclusions about whether a treatment works and a quantitive assessment of the extent to which it works.² Purchasing organisations are now being exhorted to make health policy and purchasing decisions in terms of evidence of clinical effectiveness.^{3 4} A large proportion of such evidence comes in the form of randomised controlled trials. Research has shown that the way in which the results from controlled trials are expressed has a significant influence on physicians' willingness to prescribe drugs^{5 6 7 8 9 10} and patients' perception of benefit from treatment.¹¹ Whether or not the framing of the results of controlled trials has an influence on purchasing and health policy decisions has not been examined.

There are at least four ways of reporting outcomes of clinical trials: as a relative risk reduction, absolute risk reduction, proportion of event free patients, and the number of patients who need to be treated to prevent an event.¹² Unfortunately, all too often when the results of randomised controlled trials are reported only one summary measurement of efficacy is used, most commonly the relative risk reduction. The problem with this approach is that relative risk measurements give the reader no idea of the background event rate--that is, the susceptibility of the population to the outcome of interest. Thus, when relative benefits are considerable and the outcome of interest rare, the relative risk reduction may remain high, although the absolute value of treatment will fall substantially. Quite often this problem is confounded in secondary reports and subsequent editorials that also emphasise relative differences at the expense of absolute benefits. As Feinstein states "clinicians are much impressed by the bigger numbers of relative changes than by the smaller magnitudes of the absolute changes for the same results."¹³

With the advent of evidence based medicine, understanding the quantitive methods of data presentation from clinical trials is of major importance in terms of informed decision making in health policy.² Our aim was to assess whether health policy decisions are influenced by the way in which the results of clinical trials and systematic reviews are presented.

Methods

We sent a postal questionnaire to the 182 executive and non-executive members of health authorities, health commissions, and family health services authorities within the Anglia and Oxford regional health authorities. These authorities are responsible for the setting of health policy for their local populations and are made up of people with a broad range of skills. Executive members have responsibility for purchasing, financial, and personnel management, and directors of public health have specific training in epidemiology and include in their responsibilities the assessment of the effectiveness of medical interventions and the needs of their local population. Non-executive members, most of whom are non-medical, are appointed by the chairperson of the regional health authority on behalf of the Secretary of State to act as public representatives for their local population.

The questionnaire was constructed to present data from a single clinical trial on the efficacy of breast cancer screening¹⁴ and of a single systematic review on the efficacy of cardiac rehabilitation.¹⁵ For each programme, data were presented in four different ways in the following order: as a relative risk reduction, an absolute risk reduction, as the proportion of event free patients, and as the number of patients who needed to be treated to prevent one death.

The absolute risk reduction (absolute difference) is (x-y) where x is the proportion of patients suffering the outcome of interest in the control group and y is the proportion of patients suffering the same outcome in the treatment group.¹⁶ The outcome in the control group, x, thus defines the background risk. The number of patients who need to be treated to prevent one death is the reciprocal of the absolute risk difference, 1/(x-y), and the relative risk reduction (relative difference) is [(x-y)/x]x100. The proportion of event free patients is the percentage of patients in each group, x and y, expressed in terms of survival rather than death. Table I shows a worked example from a systematic review that examined the efficacy of coronary artery bypass grafting¹⁷ in terms of survival benefit to illustrate the various methods of data presentation.

TABLE I--Method used to present the results of a randomised controlled
trial or systematic review. (Calculations adapted from Guyatt et al16;
example taken from Yusef et al17)
-----------------------------------------------------------------------------------
                                        Outcome at 10 years  Total No of patients
-----------------------------------------------------------       randomised
Treatment                                Death   Survival       in each group
-----------------------------------------------------------------------------------
Coronary artery bypass grafting           350      974              1325
Medical therapy                           404      921              1324
-----------------------------------------------------------------------------------
Risk of death at 10 years without therapy--medical treatment (background
 risk): x--404/1324=0.305 or 30.5%
Risk of death at 10 years with therapy--coronary artery bypass grafting (risk
 with therapy): y--350/1325=0.264 or 26.4%
Relative risk: y/x 0.87
Relative risk reduction: [(x-y)/x]x100 or [1-(y/x)x100=13.3%
Absolute risk reduction (or risk difference): x-y--0.04
Event free patients is absolute percentage in each group living at 10 years:
 for background risk group is (1-x)x100. For therapy group is (1-yx100).
Individuals allocated to medical treatment=69.5%
Individuals allocated to coronary artery bypass grafting=73.6%
Number needed to be treated to prevent one death is 1/(x-y)--1/0.04=25

Each set of data was presented on the questionnaire as the result of a different "trial" (appendix). Respondents were asked to mark an "X" along a linear scale marked from 0 ("I would not support the purchasing of this service") to 10 ("I would strongly support the purchasing of this service") in terms of a health authority decision to fund a breast screening programme and a cardiac rehabilitation programme. Respondents were not asked to compare the two programmes with each other. It was stated clearly that the decision to fund each programme should be taken only on the strength of the results presented from each "trial" and that costs of each alternative were the same. Respondents were asked to disregard any national or local policies with regard to the funding of either programme.

Mean scores (95% confidence interval) were calculated for each alternative presentation of results. For each programme, repeated measures analysis of variance was performed with the MANOVA procedure of SPSS for Windows.¹⁸ The global test for differences between the four options was carried out followed by pairwise multiple comparisons with the Bonferroni correction. All these analyses make full allowance for within person effects.

Results

Overall, 140 questionnaires were returned, three people stating that they could not give an opinion because they required further information. The response rate was therefore 75%, 79% for executive and 76% for non-executive members, and ranged from 68% to 94% across different health authorities.

Table II gives the descriptive statistics for the six pairwise comparisons between the options. The decision to fund either programme was significantly influenced by the way in which the data were presented (table III). The overall F test for differences between the four methods of data presentation was highly significant (P<0.001). Reporting as a relative risk reduction produced the highest mean score to support investment in both programmes, and this was significantly higher when compared with all other methods of data presentation. In addition, the number needed to treat was significantly higher than the absolute risk reduction and the percentage event free for mammography and higher than just the latter for cardiac rehabilitation (table III). Only three respondents (all non-executive members claiming no training in epidemiology) stated that they realised that all four sets of data summarised the same results.

TABLE II--Descriptive statistics for pairwise differences between four
options for each programme
----------------------------------------------------------------------------
Comparison of          Mean (SD) difference for   Mean (SD) difference for
four options               mammography             cardiac rehabilitation
----------------------------------------------------------------------------
ARR-RRR                   -4.10 (3.23)                 -1.96 (2.78)
EFP-RRR                   -4.12 (3.04)                 -4.12 (3.04)
NNT-RRR                   -2.84 (2.80)                 -1.34 (2.61)
EFP-ARR                   -0.022 (1.57)                -0.33 (1.74)
NNT-ARR                    1.26 (2.92)                  0.062 (2.40)
NNT-EFP                    0.95 (2.45)                  1.28 (2.83)
----------------------------------------------------------------------------
RRR=relative risk reduction; ARR=absolute risk reduction; EFP=event
free patients; NNT=number needed to treat.

TABLE III--Mean score (%) (95% confidence interval) on two health policy issues made by health authority
 members when same results presented by four different methods
----------------------------------------------------------------------------------------------------------
Method of data presentation             Mammography                      Cardiac rehabilitation
----------------------------------------------------------------------------------------------------------
Option A--relative risk
 reduction (RRR)                  79 (76 to 83); (RRR=34%)               76 (72 to 80); (RRR=20%)
Option B--absolute risk
 reduction (ARR)                  38 (35 to 42)+++; (ARR=0.06%)          56 (53 to 60)+; (ARR=3%)
Option C--percentages of
 event free patients (EFP)     38 (34 to 42+++; (EFP=99.82% v 99.8%) 53 (49 to 57)+++; (EFP=84% v 87%)
Option D--number needed
 to treat (NNT)                   51 (47 to 55)+; (NNT=1592)             62 (58 to 66)+; (NNT=31)
----------------------------------------------------------------------------------------------------------
*Higher score=stronger support for the purchasing of the programme.
Bonferroni difference means: +significant difference at P</=0.05 from relative risk reduction; ++significant difference
at P</=0.05 from number needed to treat.

Discussion

This study shows that interpretation of clinical trials and systematic reviews is influenced by the method used to summarise the results. Health policy makers, like clinicians,^{5 6 7 8 9} are more impressed by measurements that report relative benefits compared with those entailing an index of absolute benefit. The mean estimate obtained with the relative risk reduction for both scenarios was distinctly higher than the mean obtained with all other methods. Furthermore, such differences are most extreme when the background susceptibility--in this example, death from breast cancer--is lowest (table III).

The two scenarios were chosen deliberately to reflect the fact that absolute benefits of interventions can differ considerably, even though the relative benefits may be roughly similar. The background risk of death is 85 times higher in patients who have already suffered a myocardial infarct (background risk of death 16% over three years) compared with middle aged women who are eligible to enter a breast screening programme (background risk of death from breast cancer 0.19% over seven years), Such large absolute differences in benefit are not reflected in the results of trials when relative differences are reported alone.

Consequently, it is essential that quantitative data presented in trials and systematic reviews reflect absolute as well as relative benefits.¹⁹ This is particularly important in health policy, where background risks vary considerably according to whether interventions are aimed at populations with or without established disease. The number needed to treat possesses useful properties as a versatile way of presenting data. Calculation is relatively straightforward, background risk is incorporated into the calculation, and methods are now available to adjust it according to background risk in the target population.²⁰ Estimation of numbers needed to treat is also useful when the efficacy of different forms of interventions, such as therapeutic trials, surgical procedures, and screening programmes are compared.¹² Associated efforts, costs, and side effects of different forms of interventions can then be measured in a common currency. Furthermore, in the absence of information about the costs of alternative programmes the number needed to treat is more realistic as a proxy for cost effectiveness than is relative risk reduction.

Other commentators have suggested that the results of randomised trials and meta-analyses may require different forms of presentation when benefits at a health policy level compared with an individual level are assessed. For example, a systematic review of survival benefits from coronary artery bypass grafting on patients with stable angina expressed results in two ways.¹⁷ Firstly, as an aggregate mean survival benefit of 4.26 months at 10 years for all patients randomised to bypass grafting; and, secondly, as a disaggregated benefit (results broken down by subgroups within those randomised to bypass grafting) in which 16% of patients gained between 1.5 and 3.5 years whereas the remaining 84% had no change in life expectancy.²¹ Again, there is evidence to suggest that doctors are more impressed when the associated gain in life expectancy is expressed in a disaggregated form rather than an albeit equivalent, average figure.²² What should be recognised from all these examples^{5 6 7 8 9 10 22} is that informed clinical and health policy decisions should be made on the basis of the full understanding and limitations of the data presented. Such decisions depend on scientific description and substantive perception, not on statistical probability.¹³

Purchasing authorities are being exhorted to increase investment in clinically effective interventions and reduce investment in interventions that evidence has shown to be ineffective.²³ With the establishment of the NHS research and development programme an increasing quantity of research is likely to be presented to policy makers in the form of clinical trials and meta-analyses.⁴ In particular, Cochrane Collaboration systematic reviews and publications from the NHS Centre for Reviews and Dissemination require an understanding of the quantitive and qualitative methods used in summarising evidence. Understanding the advantages and limitations of data presentation is fundamental to the proper interpretation of such research.

This study has shown that understanding on the part of policy makers about how results of controlled trials and systematic reviews are presented needs to be enhanced. The critical appraisal skills for purchasers (CASP) project aims to help purchasers and other decision makers develop their skills in critically appraising evidence about effectiveness.²⁴ Results from this study suggest that health authorities need to consider ways in which the critical appraisal skills of their purchasers might be improved.

We would like to thank David Sackett, professor of clinical epidemiology, Centre for Evidence Based Medicine, John Radcliffe Hospital, University of Oxford, and Dr Ruairidh Milne, consultant in public health medicine, Anglia and Oxford Regional Health Authority, for their helpful suggestions on earlier drafts.

Source of funding: None.

Conflict of interest: None.

Appendix

MAMMOGRAPHY QUESTIONNAIRE

There is a proposal to offer a breast screening programme to women aged 50-64 in your health authority. There are doubts about the effectiveness of such a screening programme. We will give you four statements derived from four different randomised controlled trials published in medical journals. On the basis of each statement you should indicate how likely you are to agree to the implementation of a breast screening programme. Assume that the costs of each programme are the same. Each result was deemed to be significant.

During a seven year follow up:

* Programme A reduced the rate of deaths from breast cancer by 34%

* Programme B produced an absolute reduction in deaths from breast cancer of 0.06%

* Programme C increased the rate of patients surviving breast cancer from 99.82% to 99.88%

* Programme D meant that 1592 women needed to be screened to prevent one death from breast cancer.

CARDIAC REHABILITATION QUESTIONNAIRE

There is a proposal to offer a cardiac rehabilitation programme to people who have suffered a heart attack (myocardial infarct) in your health authority. There are doubts about the effectiveness of such a rehabilitation programme. We will give you four statements derived from four different randomised controlled trials published in medical journals. On the basis of each statement you should indicate how likely you are to agree to the implementation of a cardiac rehabilitation programme. Assume that the costs of each programme are the same. Each result was deemed to be significant.

During a three year follow up:

* Programme A reduced the rate of deaths by 20%

* Programme B produced an absolute reduction in deaths of 3%

* Programme C increased the rate of patient survival from 84% to 87%

* Programme D meant that 31 people needed to enter a rehabilitation programme to prevent one death.

1. Sackett DL, Haynes RB, Guyatt GH, Tugwell P. Clinical epidemiology: a basic science for clinical medicine. 2nd ed. Boston: Little, Brown, 1991.
2. Sackett DL, Cook RJ. Understanding clinical trials. BMJ 1994;309:755-6. [Free Full Text]
3. NHS Management Executive. Improving clinical effectiveness. Leeds: NHS Management Executive, 1993. (EL(93)115.)
4. Ham C, Hunter DJ, Robinson R. Evidence based policymaking. BMJ 1995;310:71-2. [Free Full Text]
5. Laupacis A, Sackett DL, Roberts RS. Therapeutic priorities of Canadian internists. Can Med Assoc J 1990;142:329-33. [Abstract]
6. Naylor CD, Chen E, Strauss B. Measured enthusiasm: does the method of reporting trial results alter perception of therapeutic effectiveness? Ann Intern Med 1992;117:916-21.
7. Forrow L, Taylor W, Arnold R. Absolutely relative: how research results are summarised can affect treatment decisions. Am J Med 1992;92:121-4. [Medline]
8. Farkouh ME, Sauve JS, Kassam H, Lee HN, Sackett DL. Varying formats in which trial results are reported can affect clinical decision making [abstract]. Clin Res 1993;41:517.
9. Bobbio M, Demichelis B, Giustetto G. Completeness of reporting trial results: effect on physicians' willingness to prescribe. Lancet 1994;343:1209-11. [Medline]
10. Bucher HC, Weinbacher M, Gyr K. Influence of method of reporting study results on decision of physicians to prescribe drugs to lower cholesterol concentration. BMJ 1994;309:761-4. [Abstract/Free Full Text]
11. Malenka DJ, Baron JA, Johansen S, Wahrenberger JW, Ross JM. The framing effect of relative and absolute risk. J Gen Intern Med 1993;8:543-8. [Medline]
12. Laupacis A, Sackett DL, Roberts RS. An assessment of clinically useful measures of the consequences of treatment. N Engl J Med 1988;318:1728-33. [Medline]
13. Feinstein AR. Invidious comparisons and unmet clinical challenges. Am J Med 1992;92:117-20. [Medline]
14. Tabar L, Fagerberg G, Gad A, Baldetorp L, Holmeberg LH, Grontoft O, et al. Reductions in mortality from breast cancer after mass screening with mammography. Randomised trial from the Breast Cancer Screening Group of the Swedish National Board of Health and Welfare. Lancet 1985;i:829-32.
15. Oldridge NB, Guyatt GH, Fischer ME, Rimm AA. Cardiac rehabilitation after myocardial infarction: combined experience of randomized controlled trials. JAMA 1988;260:945-50. [Abstract]
16. Guyatt GH, Sackett DL, Cook DJ, for the Evidence-Based Working Group. Users' guides to the medical literature, 11 How to use an article about therapy or prevention, B: what are the results and will they help me in caring for my patients? JAMA 1994;271:59-63.
17. Yusef S, Zucker D, Peduzzi P, Fisher LD, Takaro T, Kennedy WJ, et al. Effect of coronary artery bypass surgery on survival: overview of 10-year results from randomised trials by the coronary artery bypass graft surgery trialists collaboration. Lancet 1994;344:563-70. [Medline]
18. SPSS. SPSS for windows. Base system user's guide, release 6.0. Chicago: SPSS, 1993.
19. Laupacis A, Naylor CD, Sackett DL. How should the results of clinical trials be presented to clinicians? [abstract]. American College of Physicians Journal Club 1992;May/Jun:12-4.
20. Cook RJ, Sackett DL. The number needed to treat: a clinically useful measure of treatment. BMJ 1995;310:452-4. [Free Full Text]
21. Hux J, Naylor CD. Effect of coronary artery bypass surgery on survival [letters]. Lancet 1994;344:1223-4.
22. Hux JE, Levington CM, Naylor CD. Prescribing propensity: influence of life-expectancy gains and drug costs. J Gen Intern Med 1994;9:195-201. [Medline]
23. Hayward J. Purchasing clinically effective care. BMJ 1994;309:823-4. [Free Full Text]
24. Institute of Health Sciences. Critical appraisal skills for purchasers. Oxford: IHS, 1994. (Report to the NHS Executive.)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Related Articles

Ratio measures in leading medical journals: structured review of accessibility of underlying absolute risks

Lisa M Schwartz, Steven Woloshin, Evan L Dvorin, and H Gilbert Welch
BMJ 2006 333: 1248. [Abstract] [Full Text] [PDF]

Content of invitations for publicly funded screening mammography

Karsten Juhl Jørgensen and Peter C Gøtzsche
BMJ 2006 332: 538-541. [Full Text] [PDF]

Improving services with informatics tools

Frank Sullivan and Jeremy C Wyatt
BMJ 2005 331: 1190-1192. [Extract] [Full Text] [PDF]

Perception of risk is affected by presentation

Marion H Hall
BMJ 1995 311: 1229. [Extract] [Full Text]

This article has been cited by other articles:

• Schwartz, L. M, Woloshin, S., Dvorin, E. L, Welch, H G. (2006). Ratio measures in leading medical journals: structured review of accessibility of underlying absolute risks. BMJ 333: 1248-1248 [Abstract] [Full text]  
• Geddes, J. (2006). Providing the best available evidence: INVITED COMMENTARY ON... THE ANTIDEPRESSANT TALE. Adv. Psychiatr. Treat. 12: 327-328 [Abstract] [Full text]  
• Jorgensen, K. J., Gotzsche, P. C (2006). Content of invitations for publicly funded screening mammography.. BMJ 332: 538-541 [Full text]  
• Sullivan, F., Wyatt, J. C (2005). Improving services with informatics tools. BMJ 331: 1190-1192 [Full text]  
• Kristiansen, I. S., Fineberg, S., Kristiansen, I. S., Christensen, P. M., Gyrd-Hansen, D., Visscher, D. W., Degnim, A. C., Hartmann, L. C., Elmore, J. G., Gigerenzer, G. (2005). Benign breast disease and breast cancer.. NEJM 353: 1856-1858 [Full text]  
• Halvorsen, P. A., Kristiansen, I. S. (2005). Decisions on Drug Therapies by Numbers Needed to Treat: A Randomized Trial. Arch Intern Med 165: 1140-1146 [Abstract] [Full text]  
• Barratt, A., Wyer, P. C., Hatala, R., McGinn, T., Dans, A. L., Keitz, S., Moyer, V., for, G. G., The Evidence-Based Medicine Teaching Tips Working, (2004). Tips for learners of evidence-based medicine: 1. Relative risk reduction, absolute risk reduction and number needed to treat. CMAJ 171: 353-358 [Full text]  
• Marshall, T., Mohammed, M. A., Rouse, A. (2004). A randomized controlled trial of league tables and control charts as aids to health service decision-making. Int J Qual Health Care 16: 309-315 [Abstract] [Full text]  
• Adily, A, Ward, J (2004). Evidence based practice in population health: a regional survey to inform workforce development and organisational change. J. Epidemiol. Community Health 58: 455-460 [Abstract] [Full text]  
• Pitches, D., Burls, A., Fry-Smith, A. (2003). How to make a silk purse from a sow's ear--a comprehensive review of strategies to optimise data for corrupt managers and incompetent clinicians. BMJ 327: 1436-1439 [Full text]  
• Neves-e-Castro, M. (2003). Menopause in crisis post-Women's Health Initiative? A view based on personal clinical experience. Hum Reprod 18: 2512-2518 [Abstract] [Full text]  
• Heller, R. F, Buchan, I., Edwards, R., Lyratzopoulos, G., McElduff, P., Leger, S. S. (2003). Communicating risks at the population level: application of population impact numbers. BMJ 327: 1162-1165 [Full text]  
• Domenighetti, G., D'Avanzo, B., Egger, M., Berrino, F., Perneger, T., Mosconi, P., Zwahlen, M. (2003). Women's perception of the benefits of mammography screening: population-based survey in four countries. Int J Epidemiol 32: 816-821 [Abstract] [Full text]  
• Gigerenzer, G., Edwards, A. (2003). Simple tools for understanding risks: from innumeracy to insight. BMJ 327: 741-744 [Full text]  
• Badrinath, P, Wakeman, A. P, Wakeman, J. G, Yudkin, J. S, Parmar, M. S, Twisselmann, B. (2002). Preventing stroke with ramipril. BMJ 325: 439-439 [Full text]  
• Nexoe, J., Gyrd-Hansen, D., Kragstrup, J., Kristiansen, I. S., Nielsen, J. B. (2002). Danish GPs' perception of disease risk and benefit of prevention. Fam Pract 19: 3-6 [Abstract] [Full text]  
• Wells, J., Marshall, P., Crawley, B., Dickersin, K. (2001). Newspaper Reporting of Screening Mammography. ANN INTERN MED 135: 1029-1037 [Abstract] [Full text]  
• Ebrahim, S. (2001). The Use of Numbers Needed to Treat Derived from Systematic Reviews and Meta-Analysis: Caveats and Pitfalls. Eval Health Prof 24: 152-164 [Abstract]  
• Moynihan, R., Bero, L., Ross-Degnan, D., Henry, D., Lee, K., Watkins, J., Mah, C., Soumerai, S. B. (2000). Coverage by the News Media of the Benefits and Risks of Medications. NEJM 342: 1645-1650 [Abstract] [Full text]  
• Kee, F. (2000). Confidence limits and the limits of confidence. QJM 93: 121-124 [Full text]  
• Smeeth, L., Haines, A., Ebrahim, S. (1999). Numbers needed to treat derived from meta-analyses---sometimes informative, usually misleading. BMJ 318: 1548-1551 [Full text]  
• Mancini, G. B. J., Schulzer, M. (1999). Reporting Risks and Benefits of Therapy by Use of the Concepts of Unqualified Success and Unmitigated Failure : Applications to Highly Cited Trials in Cardiovascular Medicine. Circulation 99: 377-383 [Abstract] [Full text]  
• Bunker, J. P, Houghton, J., Baum, M. (1998). Putting the risk of breast cancer in perspective. BMJ 317: 1307-1309 [Full text]  
• Slaytor, E. K, Ward, J. E (1998). How risks of breast cancer and benefits of screening are communicated to women: analysis of 58 pamphlets. BMJ 317: 263-264 [Full text]  
• Skolbekken, J. A. (1998). Communicating the risk reduction achieved by cholesterol reducing drugs. BMJ 316: 1956-1958 [Full text]  
• McCartney, E., van der Gaag, A. (1996). How shall we be judged? Speech and language therapists in educational settings. Child Language Teaching and Therapy 12: 314-327 [Abstract]  
• Tonks, A., Smith, R. (1996). Information in practice. BMJ 313: 438-438 [Full text]  
• Hall, M. H (1995). Perception of risk is affected by presentation. BMJ 311: 1229-1229 [Full text]