BMJ 1999;319:652-653 ( 11 September )

Editorials

Can it work? Does it work? Is it worth it?

The testing of healthcare interventions is evolving

General practice p   676

The British pioneer clinical epidemiologist Archie Cochrane defined three concepts related to testing healthcare interventions.¹ Efficacy is the extent to which an intervention does more good than harm under ideal circumstances ("Can it work?"). Effectiveness assesses whether an intervention does more good than harm when provided under usual circumstances of healthcare practice ("Does it work in practice?"). Efficiency measures the effect of an intervention in relation to the resources it consumes ("Is it worth it?"). Trials of efficacy and effectiveness have also been described as explanatory and management trials, respectively,² and efficiency trials are more often called cost effectiveness or cost benefit studies.

Almost all clinical trials assess efficacy. Such trials typically select patients who are carefully diagnosed; are at highest risk of adverse outcomes from the disease in question; lack other serious illnesses; and are most likely to follow and respond to the treatment of interest. This treatment will be prescribed by doctors who are most likely to follow a careful protocol; the comparison will be a placebo, not the current best alternative therapy; and participants will receive special attention from staff who supplement or replace those employed in usual clinical settings. The results of such trials are very useful: if the intervention doesn't work under such ideal conditions it surely won't work under usual conditions. Most treatments don't survive this stage of testing, and it makes good sense to sequence the testing of all interventions through this efficacy stage.

Even if an intervention works astonishingly well in a "Can it work?" study, it may not work well in usual care. Effectiveness in the community depends not only on efficacy but also on diagnostic accuracy, provider compliance, patient adherence, and the coverage of health services.³ Misdiagnosis can result in the wrong people getting or not getting the treatment. Providers often fail to prescribe or administer the treatment properly. Patients typically take less than half of prescribed treatments. "High tech," expensive, or new interventions are usually not available in all communities in the developed countries or to most communities in the rest of the world. To paraphrase Muir Gray, what works well at the Sloan Kettering (a high tech cancer centre) may not work very well in Kettering (a small UK community).

The study by Llewellyn-Jones et al in this issue of the BMJ reveals many of these problems (p 676).⁴ In attempting to provide an intervention to help general practitioners to detect and care for depressed elderly people in residential care, the authors found little evidence that general practitioners improved their prescribing habits. Many patients refused to participate or dropped out after entry to the study. The result was a barely detectable benefit, even among those patients who stuck with the programme. Even then, the small benefit was at the expense of additional resourcesthat is, the investigators and their educational programme.

Alas, there are more troubles here. Though this study was intended to be "community based," this desired state was compromised by the difficulty of recruiting doctors and patients and keeping those recruited engaged. In the end, the study doesn't inform us about whether the community's mental health was improved. Sadly, the multiple barriers to doing health services research and implementing innovative health services are why so few investigators try to do effectiveness studies. And even if they succeed, healthcare managers, planners, and politicians will want to know more than "Does it work?": they will want to know "Is it worth it?"in comparison with use of the resources for other needs.

But don't despair. We're simply going through an evolutionary phase in testing interventions. Since the end of the second world war we've learned to walk, with randomised trials that assess efficacy. Trials such as the one by Llewellyn-Jones et al show that we're just now learning to runwith community trials that tackle difficult challenges in research design and implementation that can undermine the feasibility of a study or prejudice the interpretation of its findings. Issues of economic analysis also are being resolved, so that questions of efficiency can be better addressed. This progress will seem slow to researchers caught up in it and to all of us waiting for the answers, but in the history of the world we're heading for success at a blistering pace. Our progress is fuelled by efficacy studies and by researchers and governments intent on reaping the benefits they promise.

We need more effectiveness studies to sort the fool's gold from the true gold and efficiency studies to tell us if the price of extraction is a bargain. Fortunately, many governments around the world are aware of the need for more and better research into health services and are providing funds for training and research development. One hopes that they will not lose heart or patience: we're going in the right direction, but trial and error are needed, along with investment in methodological research to get effectiveness and efficiency studies right.

Brian Haynes, professor of clinical epidemiology and medicine. 

McMaster University Health Sciences Center, Hamilton, Ontario L8N 3Z5, Canada

---

1.	Cochrane AL. Effectiveness and efficiency: random reflection on health services. London: Nuffield Provincial Hospitals Trust , 1972.
2.	Sackett DL, Gent M. Controversies in counting and attributing events in clinical trials. N Engl J Med 1979; 301: 1410-1412[Medline].
3.	Bennett K, Tugwell P, Sackett D, Haynes B. Relative risks, benefits, and costs of intervention. In: Warren KS, Mahmoud AAF, eds. Tropical and geographic medicine. 2nd ed. New York: McGraw-Hill, 1990:205-228.
4.	Llewellyn-Jones RH, Baikie KA, Smithers H, Cohen J, Snowdon J, Tennant CC. Multifaceted shared care intervention for late life depression in residential care: randomised controlled trial. BMJ 1999; 319: 676-682[Abstract/Free Full Text].

---

© BMJ 1999
CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Related Articles

Emergency contraception

Anna Glasier
BMJ 2006 333: 560-561. [Extract] [Full Text] [PDF]

Intervention for late life depression in residential care

Ian Cameron, David Ames, Penelope Hawe, Ian McDonald, and Kerr L White
BMJ 2000 320: 119. [Extract] [Full Text]

An ABC of complementary medicine: a new dawn
BMJ 1999 319: 0. [Full Text] [PDF]

An ABC of complementary medicine: a new dawn
BMJ 1999 319: 0. [Full Text] [PDF]

Multifaceted shared care intervention for late life depression in residential care: randomised controlled trial • Commentary: Beyond the boundary for a randomised controlled trial?

Robert H Llewellyn-Jones, Karen A Baikie, Heather Smithers, Jasmine Cohen, John Snowdon, Chris C Tennant, Jonathan J Deeks, and Edmund Juszczak
BMJ 1999 319: 676-682. [Abstract] [Full Text] [PDF]

This article has been cited by other articles:

• Will, E. (2007). Intention and outcome in guideline-based nephrological practice: a suitable space for 'clinical technology'. Nephrol Dial Transplant 22: 3110-3114 [Full text]  
• Gilliam, M. (2006). Emergency contraception: politics and science move forward.. Obstet Gynecol 108: 1060-1061 [Full text]  
• Helmerhorst, F.M., Perquin, D.A.M., Dorr, P.J., de Craen, A.J.M. (2006). Reply: Routine use of hysterosalpingography prior to diagnostic laparoscopy in the fertility workup. Hum Reprod 21: 2726-2727 [Full text]  
• Glasier, A. (2006). Emergency contraception. BMJ 333: 560-561 [Full text]  
• Perquin, D.A.M., Dorr, P.J., de Craen, A.J.M., Helmerhorst, F.M. (2006). Routine use of hysterosalpingography prior to laparoscopy in the fertility workup: a multicentre randomized controlled trial. Hum Reprod 21: 1227-1231 [Abstract] [Full text]  
• Chren, M.-M. (2006). Determining the value of surgical therapies for Basal cell carcinoma.. Arch Dermatol 142: 231-232 [Full text]  
• Eijkemans, M.J.C., Heijnen, E.M.E.W., de Klerk, C., Habbema, J.D.F., Fauser, B.C.J.M. (2006). Comparison of different treatment strategies in IVF with cumulative live birth over a given period of time as the primary end-point: methodological considerations on a randomized controlled non-inferiority trial. Hum Reprod 21: 344-351 [Abstract] [Full text]  
• Collins, J. (2005). Which Randomized Controlled Trials Are Relevant to Clinical Practice?. Obstet Gynecol 106: 216-218 [Full text]  
• Arce, J.-C., Nyboe Andersen, A., Collins, J. (2005). Resolving methodological and clinical issues in the design of efficacy trials in assisted reproductive technologies: a mini-review. Hum Reprod 20: 1757-1771 [Abstract] [Full text]  
• Atkins, D., Siegel, J., Slutsky, J. (2005). Making Policy When The Evidence Is In Dispute. Health Aff (Millwood) 24: 102-113 [Abstract] [Full text]  
• Alderson, P., Groves, T. (2004). What doesn't work and how to show it. BMJ 328: 473-473 [Full text]  
• Torres, A., Muir, J-F., Corris, P., Kubin, R., Duprat-Lomon, I., Sagnier, P-P., Hoffken, G. (2003). Effectiveness of oral moxifloxacin in standard first-line therapy in community-acquired pneumonia. Eur Respir J 21: 135-143 [Abstract] [Full text]  
• Sim, I., Gorman, P., Greenes, R. A., Haynes, R. B., Kaplan, B., Lehmann, H., Tang, P. C. (2001). Clinical Decision Support Systems for the Practice of Evidence-based Medicine. J. Am. Med. Inform. Assoc. 8: 527-534 [Abstract] [Full text]  
• Allan, L., Hays, H., Jensen, N.-H., de Waroux, B. L. P., Bolt, M., Donald, R., Kalso, E. (2001). Randomised crossover trial of transdermal fentanyl and sustained release oral morphine for treating chronic non-cancer pain. BMJ 322: 1154-1154 [Abstract] [Full text]  
• Robertson, M C., Gardner, M. M, Devlin, N., McGee, R., Campbell, A J. (2001). Effectiveness and economic evaluation of a nurse delivered home exercise programme to prevent falls. 2: Controlled trial in multiple centres. BMJ 322: 701-701 [Abstract] [Full text]  
• Will, E J (2001). Aiming at averages. JRSM 94: 617-619 [Full text]  
• Greenhalgh, T. (2001). Computer assisted learning in undergraduate medical education. BMJ 322: 40-44 [Full text]  
• Vickers, A. (2000). Recent advances: Complementary medicine. BMJ 321: 683-686 [Full text]  
• Satya-Murti, S. (2000). Clinical Evidence: A Compendium of the Best Available Evidence for Effective Health Care. JAMA 283: 1352-1353 [Full text]  
• Will, E. J., Cameron, J. S. (2000). European guidelines for renal anaemia--predicting 85% compliance. Nephrol Dial Transplant 15: 439-440 [Full text]  
• Cameron, I., Ames, D., Hawe, P., McDonald, I., White, K. L (2000). Intervention for late life depression in residential care. BMJ 320: 119a-119 [Full text]  
• Llewellyn-Jones, R. H, Baikie, K. A (1999). Intervention trial for late life depression defended. BMJ 319: 1068-1068 [Full text]  

Rapid Responses:

Read all Rapid Responses

Intervention trial for late life depression defended

Robert H Llewellyn-Jones, et al.
bmj.com, 24 Sep 1999 [Full text]

Terms of reference

Ian Macdonald
bmj.com, 1 Oct 1999 [Full text]

How Much Trial and Error Should We Tolerate in Community Trials?

Penny Hawe
bmj.com, 1 Oct 1999 [Full text]

Efficacy & Effectivness: Origins of the Concepts & Terms

Kerr L White
bmj.com, 18 Oct 1999 [Full text]