Can it work? Does it work? Is it worth it?BMJ 1999; 319 doi: https://doi.org/10.1136/bmj.319.7211.652 (Published 11 September 1999) Cite this as: BMJ 1999;319:652
All rapid responses
Brian Haynes credits Archie Cochrane with first defining the terms
“efficacy, “effectiveness, and “efficiency” as applied to health services
(BMJ 1999;319:652-653, September 11). Archie did much to popularize the
application of the terms but they were first promulgated two years earlier
at the 14th meeting of the WHO Expert Committee on Health Statistics,
The late Sir John Brotherston was in the chair and I, as the
rapporteur, wrote the report. The concepts and terms, however, came from
two other members of the Committee: Dr. A. Sakari Härö, Chief, Department
of Planning, National Board of Health, Helsinki, Finland, and the late
Dr.Georges Rösch, Deputy Director, Centre de Recherches et de
Documentation sur la Consommation, Paris, France. The definitions were:
Efficacy: the benefit or utility to the individual of the service,
treatment regimen, drug, preventive or control measure advocated or
Effectiveness: the effect of the activity and the end-results,
outcomes or benefits for the population achieved in relation to the stated
Efficiency: the effects or end-results achieved in relation to the
effort expended in terms of money, resources, and time.
Any member of our Committee could have discussed the terms with
Archie but the most likely candidates are John Brotherston or Dr. W.P.D.
Logan from the UK; the latter at that time was the Director of the
Division of Health Statistics at WHO. Alternatively Archie may have
conceived them independently but unfortunately he did not distinguish
clearly between the first two.
John Last is correcting the entry in the 4th edition of the "Dictionary of Epidemiology", published by Oxford University Press.
1 Statistical Indicators for the Planning amd Evaluation of Public Health Progammes: Technical Report No. 472.Geneva: World Health Organization, 1971.
Competing interests: No competing interests
Professor Haynes' editorial1 refers to a trial by Llewellyn-Jones
and colleagues where the modest result is attributed to a variable, but
"naturalistic" degree of program implementation.2 Haynes assures us that
"trial and error" is a necessary part of the evolution of trials. This
ignores the broader question of how much effort should we undertake to get
an intervention right before we put it to the ultimate test (the RCT),
particularly ambitious interventions which set out to "change the care
culture"2? When we depart from the tidy world of drug trials to the murky
world of community trials, where do we draw the line between efficacy and
effectiveness? If we are too ready to accept the "real world" conditions
of effectiveness trials, we may risk a proliferation of state-of-the-art
evaluations of far less than state-of-the-art interventions. This prospect
seems extraordinarily wasteful.
These issues are felt acutely in the health promotion, where some
spectacular failures have led to intensive soul-searching about the
differences between program failure and evaluation failure and, in the
event of program failure, whether this is accounted for by implementation
failure or theory failure. An RCT is an unnecessarily expensive way of
learning about implementation failure. This has led to the view that a
cycle of implementation and review, of getting the implementation as right
as feasibly possible, should precede the evaluation of program outcomes.
In other words, RCTs should have starting rules as well as stopping rules.
Professor Haynes argues that we are only "learning to run" with community
trials. This may be the case, but we are certainly not just learning to
run with community interventions. Nor are we ignorant of methods to assess
contextual factors in program environments,3 nor methods to guide change
processes,4 nor methods to assess implementation.5 This means we are
better equipped than ever to introduce programs and optimise their
functioning prior to testing. Undoubtedly, professional judgement is
required to determine whether implementation is "as right as feasibly
possible." What range and type of evidence and expertise should be called
upon? How, for example, might we distinguish "naturalistic" conditions
from poor program management within a trial? A lot of this territory is
uncharted because many investigators appear, at least, to pay it scant
attention. Expert criticism of intervention theory and strategy, as well
as scrutiny of the criteria to be used to define intervention integrity,
must be part of trial design and review.
Department of Public Health and Community Medicine (A27),
University of Sydney,
New South Wales 2006,
1 Haynes B. Can it work? Does it work? Is it worth it? British
Medical Journal 1999;319:653-654
2 Llewellyn-Jones RH, Baikie KA, Smithers H, Cohen J, Snowdon J,
Tennant CC. Multifaceted shared care intervention for late life depression
in residential care: randomised controlled trial. British Medical Journal
3 Moos RH. Assessing the program environment: implications for
program evaluation and design. In Conrad KJ, Roberts-Gray C. (Eds)
Evaluating Program Environments. New Directions in Program Evaluation. No
40 Jossey Bass Higher Education and Social and Behavioral Sciences
Series. San Francisco 1988
4 Goodman RM, Steckler AB. Mobilising organisations for health
enhancement: theories of organisational change. In Glanz K, Lewis FM,
Rimer BK (Eds) Health Behaviour and Health Education. Theory, Research and
Practice. Jossey Bass. San Francisco 1990
5 Lynch KB, Geller SR, Hunt DR, Galano J, Dubas JS. Successful
program development using implementation evaluation. Journal of Prevention
and Intervention in the Community 1998;17(2):51-64
Competing interests: No competing interests
In his recent editorial , Brian Haynes quoted definitions of
"efficacy" and "effectiveness" which he attributes to Archie Cochrane .
John Last has made the same attribution for his definition of
'effectiveness' . For all Cochrane's admirable achievements, this was
not the case. Referring to the need to apply the RCT used "to measure the
effect of a particular medical action in altering the natural history of a
particular disease for the better", Archie Cochrane stated: "It is in
this sense that I use the word 'effective' in this book, and I use it in
relation to research results, as opposed to the results obtained when a
therapy is used in routine practice in a defined community. Some people
would like to use the word 'efficacious' for this measurement. This seems
reasonable, but as I dislike the word I have not used it here." Hence he
used the terminology which is almost exactly the opposite of that
attributed to him.
Kerr White recounts the origin of the current widely used terminology as
follows: "The distinction between 'effectiveness' and 'efficacy' was made
originally in our 1971 Technical Report of the WHO Expert Committee on
Health Statistics. John Brotherston was chair and I was Rapporteur but
the ideas came from the late Georges Rosch, a French health economist, and
Sakari Haro, the most knowledgeable person on health statistics and health
information systems I have encountered" .
Centre for the Study of Clinical Practice,
St Vincent's Hospital Melbourne,
Fitzroy, Victoria 3065,
1 Haynes B. Can it work? Does it work? Is it worth it?: the
testing of health care interventions is evolving. BMJ 1999;319:652-3.
2 Cochrane AL. Effectiveness and efficiency: random reflections on
health services. London: Nuffield Provincial Hospitals Trust, 1972.
3 Last JM. A Dictionary of Epidemiology. 3rd Edition. Oxford:
Oxford University Press, 1995.
4 White KW. Personal communication, 1998.
Competing interests: No competing interests
EDITOR-Brian Haynes' editorial and Deeks and Juszczak's
commentary make valuable points about the testing of healthcare
interventions. However, we would like to respond to several criticisms
they have made of our study.
Haynes argues that our effectiveness study is limited by patients
refusing to participate or dropping out after study entry. We recognise
the importance of maximising recruitment and retention of participants,
but the practical difficulties need to be considered. In the context of
studies of elderly people, we consider that our refusal rates of 21% or
less and moderately high follow-up rates (at least 75% for the geriatric
depression scale and 58% for all measures) were reasonable. It is
difficult to recruit elderly people into research studies and maintain
their participation over lengthy periods. Considerable dropout rates were
to be expected over the 9.5 months of the study period because our sample
consisted of predominantly very old, frail people, many of whom died or
became too ill to participate.
Haynes is also concerned that our difficulty recruiting doctors and
patients and keeping those recruited engaged has compromised our community
-based intervention. In our paper we acknowledge the importance of
maximising participation. However, the practical difficulties of fully
engaging a whole community, particularly when it consists of busy doctors
and frail elderly people, also needs to be acknowledged. In fact, 62% of
the intervention group participants had general practitioners who attended
the educational program, and 35 (49%) of the 71 general practitioners
caring for all 1036 residents screened for study entry attended the
education. In our experience, these attendance rates are above average
for continuing medical education programs for general practitioners.
Also, although the proportion of intervention group participants attending
exercise classes (28%) may seem low, remember that the intervention sought
to reach a group of very old, frail, depressed individuals who would
generally be unlikely to attend such classes. Furthermore, by design we
did not specifically target study participants.
Finally, Haynes is concerned that the intervention's effect was "at
the expense of additional resources - that is, the investigators and their
educational program". Whilst it is true that some additional resources
were required, the intervention was not expensive to run. To make it
widely generalisable and sustainable, we deliberately used existing health
care resources wherever possible. Apart from a grant-funded 1.0 full-time
-equivalent Project Officer, the work of others involved in the
intervention was encompassed within their usual positions. Given that the
educational materials required for the intervention have now been devised,
it would be possible to implement the intervention at a similar site with
reasonable existing health care resources for the cost of this full-time
Project Officer position.
Deeks and Juszczak's commentary begins by outlining four
"deficiencies" which may have affected the study findings, however they
appear to conclude that only one of these, namely the lack of a concurrent
control group, is likely to have introduced bias. As we indicate in the
limitations section of the paper, we only abandoned these key principles
of randomised controlled trials after considerable thought because there
was no sensible, practical alternative study design. Whilst there is
undeniable value in strictly adhering to the standard randomised
controlled trial design to minimise bias, this needs to be weighed up
against the methodological difficulties and practical constraints involved
in health services research of this type.
Deeks and Juszczak also question whether the intervention may have
had a harmful effect among the non-depressed residents. Whilst we
recognise that this is theoretically possible, it is unlikely to be the
case. The intervention was specifically designed to fit in with
participants' normal work loads. In the project teams' frequent
consultations with caregivers, there was no evidence that carers were
neglecting more pressing health needs in favour of the intervention.
Our final concern is that if undue weight is given to the above
criticisms our study's important clinical message will be lost. Late life
depression is a significant public health problem. The modest but
significant intervention effect in this study was achieved by encouraging
existing health care providers to work more effectively. More substantial
outcomes would be expected with greater participation. Despite the
methodological issues raised about the study, we believe that this
multifaceted intervention is a promising way of addressing late life
depression in residential care.
1. Haynes B. Can it work? Does it work? Is it worth it? [editorial].
2. Deeks JJ, Juszczak E. Commentary: beyond the boundary for a randomised
controlled trial? BMJ 1999;319:682. www.bmj.com/cgi/content/full/319/7211/676
3. Llewellyn-Jones RH, Baikie KA, Smithers H, Cohen J, Snowdon J, Tennant
CC. Multifaceted shared care intervention for late life depression in
residential care: randomised controlled trial. BMJ 1999;319:676-682. www.bmj.com/cgi/content/full/319/7211/676
4. Cameron ID. Recruiting older people for clinical trials and health
promotion programs. Med J Aust 1997;167:441.
The "Insights" general practitioner education workshop on late life
depression that was included in the intervention was devised using an
educational grant from Roche Products. RHL-J received a small honorarium
from Roche Products for attending "Insights" Advisory Committee meetings.
RHL-J was also reimbursed by Roche Products for attending a symposium.
KAB's work on the study was part funded by a Roche Products Clinical
Research Grant. Neither the "Insights" General Practitioner Education
meetings nor the multifaceted intervention as a whole promoted the use of
any specific brand of pharmaceutical product.
Competing interests: No competing interests