Rapid Responses to:

EDITORIALS: Brian Haynes Can it work? Does it work? Is it worth it? BMJ 1999; 319: 652-653 [Full text]

Rapid Responses: Submit a response to this article

Rapid Responses published:

Intervention trial for late life depression defended

Robert H Llewellyn-Jones, Karen A Baikie   (24 September 1999)

Terms of reference

Ian Macdonald   (1 October 1999)

How Much Trial and Error Should We Tolerate in Community Trials?

Penny Hawe   (1 October 1999)

Efficacy & Effectivness: Origins of the Concepts & Terms

Kerr L White   (18 October 1999)

Intervention trial for late life depression defended 24 September 1999
Robert H Llewellyn-Jones, Lecturer; Senior Research Officer Department of Psychological Medicine, University of Sydney, New South Wales, 2006, Australia, Karen A Baikie Send response to journal: Re: Intervention trial for late life depression defended	EDITOR-Brian Haynes' editorial[1] and Deeks and Juszczak's commentary[2] make valuable points about the testing of healthcare interventions. However, we would like to respond to several criticisms they have made of our study.[3] Haynes argues that our effectiveness study is limited by patients refusing to participate or dropping out after study entry. We recognise the importance of maximising recruitment and retention of participants, but the practical difficulties need to be considered. In the context of studies of elderly people, we consider that our refusal rates of 21% or less and moderately high follow-up rates (at least 75% for the geriatric depression scale and 58% for all measures) were reasonable. It is difficult to recruit elderly people into research studies[4] and maintain their participation over lengthy periods. Considerable dropout rates were to be expected over the 9.5 months of the study period because our sample consisted of predominantly very old, frail people, many of whom died or became too ill to participate. Haynes is also concerned that our difficulty recruiting doctors and patients and keeping those recruited engaged has compromised our community -based intervention. In our paper we acknowledge the importance of maximising participation. However, the practical difficulties of fully engaging a whole community, particularly when it consists of busy doctors and frail elderly people, also needs to be acknowledged. In fact, 62% of the intervention group participants had general practitioners who attended the educational program, and 35 (49%) of the 71 general practitioners caring for all 1036 residents screened for study entry attended the education. In our experience, these attendance rates are above average for continuing medical education programs for general practitioners. Also, although the proportion of intervention group participants attending exercise classes (28%) may seem low, remember that the intervention sought to reach a group of very old, frail, depressed individuals who would generally be unlikely to attend such classes. Furthermore, by design we did not specifically target study participants. Finally, Haynes is concerned that the intervention's effect was "at the expense of additional resources - that is, the investigators and their educational program". Whilst it is true that some additional resources were required, the intervention was not expensive to run. To make it widely generalisable and sustainable, we deliberately used existing health care resources wherever possible. Apart from a grant-funded 1.0 full-time -equivalent Project Officer, the work of others involved in the intervention was encompassed within their usual positions. Given that the educational materials required for the intervention have now been devised, it would be possible to implement the intervention at a similar site with reasonable existing health care resources for the cost of this full-time Project Officer position. Deeks and Juszczak's commentary begins by outlining four "deficiencies" which may have affected the study findings, however they appear to conclude that only one of these, namely the lack of a concurrent control group, is likely to have introduced bias. As we indicate in the limitations section of the paper, we only abandoned these key principles of randomised controlled trials after considerable thought because there was no sensible, practical alternative study design. Whilst there is undeniable value in strictly adhering to the standard randomised controlled trial design to minimise bias, this needs to be weighed up against the methodological difficulties and practical constraints involved in health services research of this type. Deeks and Juszczak also question whether the intervention may have had a harmful effect among the non-depressed residents. Whilst we recognise that this is theoretically possible, it is unlikely to be the case. The intervention was specifically designed to fit in with participants' normal work loads. In the project teams' frequent consultations with caregivers, there was no evidence that carers were neglecting more pressing health needs in favour of the intervention. Our final concern is that if undue weight is given to the above criticisms our study's important clinical message will be lost. Late life depression is a significant public health problem. The modest but significant intervention effect in this study was achieved by encouraging existing health care providers to work more effectively. More substantial outcomes would be expected with greater participation. Despite the methodological issues raised about the study, we believe that this multifaceted intervention is a promising way of addressing late life depression in residential care. References 1. Haynes B. Can it work? Does it work? Is it worth it? [editorial]. BMJ 1999;319:652-653. 2. Deeks JJ, Juszczak E. Commentary: beyond the boundary for a randomised controlled trial? BMJ 1999;319:682. www.bmj.com/cgi/content/full/319/7211/676 3. Llewellyn-Jones RH, Baikie KA, Smithers H, Cohen J, Snowdon J, Tennant CC. Multifaceted shared care intervention for late life depression in residential care: randomised controlled trial. BMJ 1999;319:676-682. www.bmj.com/cgi/content/full/319/7211/676 4. Cameron ID. Recruiting older people for clinical trials and health promotion programs. Med J Aust 1997;167:441. Competing Interests The "Insights" general practitioner education workshop on late life depression that was included in the intervention was devised using an educational grant from Roche Products. RHL-J received a small honorarium from Roche Products for attending "Insights" Advisory Committee meetings. RHL-J was also reimbursed by Roche Products for attending a symposium. KAB's work on the study was part funded by a Roche Products Clinical Research Grant. Neither the "Insights" General Practitioner Education meetings nor the multifaceted intervention as a whole promoted the use of any specific brand of pharmaceutical product.
Terms of reference 1 October 1999
Ian Macdonald Send response to journal: Re: Terms of reference	In his recent editorial [1], Brian Haynes quoted definitions of "efficacy" and "effectiveness" which he attributes to Archie Cochrane [2]. John Last has made the same attribution for his definition of 'effectiveness' [3]. For all Cochrane's admirable achievements, this was not the case. Referring to the need to apply the RCT used "to measure the effect of a particular medical action in altering the natural history of a particular disease for the better", Archie Cochrane stated: "It is in this sense that I use the word 'effective' in this book, and I use it in relation to research results, as opposed to the results obtained when a therapy is used in routine practice in a defined community. Some people would like to use the word 'efficacious' for this measurement. This seems reasonable, but as I dislike the word I have not used it here." Hence he used the terminology which is almost exactly the opposite of that attributed to him. Kerr White recounts the origin of the current widely used terminology as follows: "The distinction between 'effectiveness' and 'efficacy' was made originally in our 1971 Technical Report of the WHO Expert Committee on Health Statistics. John Brotherston was chair and I was Rapporteur but the ideas came from the late Georges Rosch, a French health economist, and Sakari Haro, the most knowledgeable person on health statistics and health information systems I have encountered" [4]. Ian McDonald Director, Centre for the Study of Clinical Practice, St Vincent's Hospital Melbourne, Victoria Pde, Fitzroy, Victoria 3065, Australia 1 Haynes B. Can it work? Does it work? Is it worth it?: the testing of health care interventions is evolving. BMJ 1999;319:652-3. 2 Cochrane AL. Effectiveness and efficiency: random reflections on health services. London: Nuffield Provincial Hospitals Trust, 1972. 3 Last JM. A Dictionary of Epidemiology. 3rd Edition. Oxford: Oxford University Press, 1995. 4 White KW. Personal communication, 1998.
How Much Trial and Error Should We Tolerate in Community Trials? 1 October 1999
Penny Hawe Send response to journal: Re: How Much Trial and Error Should We Tolerate in Community Trials?	Professor Haynes' editorial1 refers to a trial by Llewellyn-Jones and colleagues where the modest result is attributed to a variable, but "naturalistic" degree of program implementation.2 Haynes assures us that "trial and error" is a necessary part of the evolution of trials. This ignores the broader question of how much effort should we undertake to get an intervention right before we put it to the ultimate test (the RCT), particularly ambitious interventions which set out to "change the care culture"2? When we depart from the tidy world of drug trials to the murky world of community trials, where do we draw the line between efficacy and effectiveness? If we are too ready to accept the "real world" conditions of effectiveness trials, we may risk a proliferation of state-of-the-art evaluations of far less than state-of-the-art interventions. This prospect seems extraordinarily wasteful. These issues are felt acutely in the health promotion, where some spectacular failures have led to intensive soul-searching about the differences between program failure and evaluation failure and, in the event of program failure, whether this is accounted for by implementation failure or theory failure. An RCT is an unnecessarily expensive way of learning about implementation failure. This has led to the view that a cycle of implementation and review, of getting the implementation as right as feasibly possible, should precede the evaluation of program outcomes. In other words, RCTs should have starting rules as well as stopping rules. Professor Haynes argues that we are only "learning to run" with community trials. This may be the case, but we are certainly not just learning to run with community interventions. Nor are we ignorant of methods to assess contextual factors in program environments,3 nor methods to guide change processes,4 nor methods to assess implementation.5 This means we are better equipped than ever to introduce programs and optimise their functioning prior to testing. Undoubtedly, professional judgement is required to determine whether implementation is "as right as feasibly possible." What range and type of evidence and expertise should be called upon? How, for example, might we distinguish "naturalistic" conditions from poor program management within a trial? A lot of this territory is uncharted because many investigators appear, at least, to pay it scant attention. Expert criticism of intervention theory and strategy, as well as scrutiny of the criteria to be used to define intervention integrity, must be part of trial design and review. Penelope Hawe, senior lecturer Department of Public Health and Community Medicine (A27), University of Sydney, New South Wales 2006, Australia 1 Haynes B. Can it work? Does it work? Is it worth it? British Medical Journal 1999;319:653-654 2 Llewellyn-Jones RH, Baikie KA, Smithers H, Cohen J, Snowdon J, Tennant CC. Multifaceted shared care intervention for late life depression in residential care: randomised controlled trial. British Medical Journal 1999;319:676-682 3 Moos RH. Assessing the program environment: implications for program evaluation and design. In Conrad KJ, Roberts-Gray C. (Eds) Evaluating Program Environments. New Directions in Program Evaluation. No 40 Jossey Bass Higher Education and Social and Behavioral Sciences Series. San Francisco 1988 4 Goodman RM, Steckler AB. Mobilising organisations for health enhancement: theories of organisational change. In Glanz K, Lewis FM, Rimer BK (Eds) Health Behaviour and Health Education. Theory, Research and Practice. Jossey Bass. San Francisco 1990 5 Lynch KB, Geller SR, Hunt DR, Galano J, Dubas JS. Successful program development using implementation evaluation. Journal of Prevention and Intervention in the Community 1998;17(2):51-64
Efficacy & Effectivness: Origins of the Concepts & Terms 18 October 1999
Kerr L White, Retired Deputy Director for Health Sciences, the Rockefeller Foundation, New York, NY 500 Crestwood Drive, #1410, Charlottesville, Virginia, USA Send response to journal: Re: Efficacy & Effectivness: Origins of the Concepts & Terms	Brian Haynes credits Archie Cochrane with first defining the terms “efficacy, “effectiveness, and “efficiency” as applied to health services (BMJ 1999;319:652-653, September 11). Archie did much to popularize the application of the terms but they were first promulgated two years earlier at the 14th meeting of the WHO Expert Committee on Health Statistics, December 1-7,1970.[1] The late Sir John Brotherston was in the chair and I, as the rapporteur, wrote the report. The concepts and terms, however, came from two other members of the Committee: Dr. A. Sakari Härö, Chief, Department of Planning, National Board of Health, Helsinki, Finland, and the late Dr.Georges Rösch, Deputy Director, Centre de Recherches et de Documentation sur la Consommation, Paris, France. The definitions were: Efficacy: the benefit or utility to the individual of the service, treatment regimen, drug, preventive or control measure advocated or applied. Effectiveness: the effect of the activity and the end-results, outcomes or benefits for the population achieved in relation to the stated objectives. Efficiency: the effects or end-results achieved in relation to the effort expended in terms of money, resources, and time. Any member of our Committee could have discussed the terms with Archie but the most likely candidates are John Brotherston or Dr. W.P.D. Logan from the UK; the latter at that time was the Director of the Division of Health Statistics at WHO. Alternatively Archie may have conceived them independently but unfortunately he did not distinguish clearly between the first two. John Last is correcting the entry in the 4th edition of the "Dictionary of Epidemiology", published by Oxford University Press. Reference 1 Statistical Indicators for the Planning amd Evaluation of Public Health Progammes: Technical Report No. 472.Geneva: World Health Organization, 1971.