BMJ  2008;336:516-517 (8 March), doi:10.1136/bmj.39510.531597.80 (published 4 March 2008)

Editorials

Efficacy of antidepressants

Is not an absolute measure, and it depends on how clinical significance is defined

In February 2008, Kirsch and colleagues reported a meta-analysis of the efficacy of antidepressants using data from clinical trials submitted to the Food and Drug Administration.¹ They provocatively concluded, "there seems little evidence to support the prescription of antidepressant medication to any but the most severely depressed patients."

In January this year, we published an article about the selective publication of antidepressant trials and its influence on apparent efficacy,² in which we also used FDA data. Our main finding was that antidepressant drugs are much less effective than is apparent from journal articles. From the FDA data we derived an overall effect size of 0.31. Kirsch and colleagues used FDA data from four of the 12 drugs we examined and calculated an overall effect size of 0.32.

Although these two sets of results were in excellent agreement, our interpretations of them were quite different. In contrast to Kirsch and colleagues’ conclusion that antidepressants are ineffective, we concluded that each drug was superior to placebo. The difference in our interpretations stems from Kirsch and colleagues’ use of the criteria for clinical significance recommended by the UK’s National Institute for Health and Clinical Excellence (NICE).

Clinical significance is an important concept because a clinical trial can show superiority of a drug to placebo in a way that is statistically, but not clinically, significant. Tests of statistical significance give a yes or no answer (for example, P<0.05 is deemed significant, P>0.05 non-significant) that tells us whether the true effect size is zero or not, but it tells us nothing about the size of the effect.³ In contrast, effect size does, and thus allows us to look at the question of clinical significance. Values of 0.2, 0.5, and 0.8 were proposed to represent small, medium, and large effects, respectively.⁴

NICE chose the "medium" value of 0.5 as a cut-off below which they deem benefit of a drug not clinically significant.⁵ This is problematic because it transforms effect size, a continuous measure, into a yes or no measure, thereby suggesting that drug efficacy is either totally present or absent, even when comparing values as close together as 0.51 and 0.49. Kirsch and colleagues compared their effect size of 0.32 to the 0.50 cut-off and concluded that the benefits of antidepressant drugs were of no clinical significance.

But on what basis did NICE adopt the 0.5 value as a cut-off? When Cohen first proposed these landmark effect size values, he wrote, "The terms ‘small’, ‘medium’, and ‘large’ are relative . . . to each other . . . the definitions are arbitrary . . . these proposed conventions were set forth throughout with much diffidence, qualifications, and invitations not to employ them if possible." He also said, "The values chosen had no more reliable a basis than my own intuition." Thus, it seems doubtful that he would have endorsed NICE’s use of an effect size of 0.5 as a litmus test for drug efficacy.

To illustrate Cohen’s use of "relative" with a metaphor, imagine antidepressant efficacy measured in terms of litres of a fluid called "d-juice" (named after Cohen’s "d"—the effect size measure described here). When our group measured 0.41 litres of d-juice in the "glass" representing journal articles, but 0.31 litres in the FDA glass, we concluded that the FDA glass was empty relative to the journal glass. Nevertheless, we acknowledged that 0.31 litres was an amount that was measurable and significant. Kirsch and colleagues measured 0.32 litres of d-juice, but because they did not consider the glass sufficiently full (defined arbitrarily as P0.5), they concluded that the glass contained virtually no d-juice whatsoever. To summarise, we agree that the antidepressant "glass" is far from full, but we disagree that it is completely empty.

Hypothetically, if antidepressants are not worth taking, then what should doctors and patients do? Kirsch and colleagues recommend that if antidepressants are to be used at all they should be used only when alternative treatments have failed to provide a benefit.¹ Although the authors did not specify a preferred first line treatment, they may have had psychotherapy in mind.^{6 7} It seems unfair that pharmacological, and not psychotherapeutic, treatment has become the usual first line approach to depression merely for economic reasons.⁷ But before we embrace any treatment as first line, it is prudent to ask whether its efficacy is beyond question. For psychotherapy trials, there is no equivalent of the FDA whose records we can examine, so how can we be sure that selective publication is not occurring here as well?

Our clinical recommendation is that when considering the potential benefits of treatment with antidepressants, be circumspect but not dismissive. Efficacy measured in clinical trials does not necessarily translate into effectiveness in clinical practice.⁸ Patients’ individual responses are like clinical trial effect sizes in that they are not all or none. Thus, when a patient is tried on his or her first antidepressant, a partial response should not be surprising or discouraging. Also, depression rating scales used in clinical trials seldom measure quality of life, which has been suggested to be a reasonable measure of clinical significance.⁹

With regard to policy, we reiterate our request in 2004 for drug regulatory authorities such as the FDA to make their reviews publicly available on the world wide web—retrospectively.¹⁰ Making this unbiased information more accessible will allow other researchers to move beyond antidepressants and ascertain the true efficacy of all marketed drugs.

¹ Department of Psychiatry, Oregon Health and Science University, Portland, OR 97239, USA, ² Department of Psychology, University of California, Riverside, CA 92521, USA

---

Competing interests: None declared.

Provenance and peer review: Commissioned; not externally peer reviewed.

References

1. Kirsch I, Deacon BJ, Huedo-Medina TB, Scoboria A, Moore TJ, Johnson BT. Initial severity and antidepressant benefits: a meta-analysis of data submitted to the Food and Drug Administration. PLoS Med 2008;5:e45.[CrossRef][Medline]
2. Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008;358:252-60.[Abstract/Free Full Text]
3. Cohen J. The earth is round (P<.05). Am Psychologist 1994;49:997-1003.[CrossRef]
4. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. New York: Lawrence Erlbaum Associates, 1988.
5. National Institute for Health and Clinical Excellence. Depression: management of depression in primary and secondary care. Clinical practice guideline CG23. 2004. www.nice.org.uk/nicemedia/pdf/cg023fullguideline.pdf.
6. Antonuccio DO, Burns DD, Danton WG. Antidepressants: a triumph of marketing over science? Prevent Treat 2002;5:Article 25.
7. Antonuccio DO, Danton WG, DeNelsky GY. Psychotherapy versus medication for depression: challenging the conventional wisdom with data. Prof Psychol Res Pract 1995;26:574-85.[CrossRef][ISI]
8. Revicki DA, Frank L. Pharmacoeconomic evaluation in the real world. Effectiveness versus efficacy studies. Pharmacoeconomics 1999;15:423-34.[CrossRef][ISI][Medline]
9. Gladis M. Quality of life: expanding the scope of clinical significance. J Consult Clin Psychol 1999;67:320-31.[CrossRef][ISI][Medline]
10. Turner EH. A taxpayer-funded clinical trials registry and results database. PLoS Med 2004;1:e60. http://dx.doi.org/10.1371/journal.pmed.0010060.[CrossRef][Medline]

Related Articles

How full is the glass?

Irving Kirsch and Blair T Johnson
BMJ 2008 336: 629-630. [Extract] [Full Text] [PDF]

Finding meanings

Jane Smith
BMJ 2008 336: 0. [Extract] [Full Text]

Bitter sweets

Des Spence
BMJ 2008 336: 562. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

• Kirsch, I., Johnson, B. T (2008). How full is the glass?. BMJ 336: 629-630 [Full text]  

Rapid Responses:

Read all Rapid Responses

Clarity triumphs over complexity

Alex J Stockdale
bmj.com, 7 Mar 2008 [Full text]

Why is the effect size so small?

D B Double
bmj.com, 8 Mar 2008 [Full text]

Depression is a "Mixed Bag"

Joseph More
bmj.com, 9 Mar 2008 [Full text]

Clarity of expression but a misleading image

Allan O House
bmj.com, 9 Mar 2008 [Full text]

what would be the destiny of people with learning disability

Rehana Shakir
bmj.com, 9 Mar 2008 [Full text]

Efficacy of Antidepressants: How Full Is the Glass?

Irving Kirsch, et al.
bmj.com, 10 Mar 2008 [Full text]

A la recherche du temps passé

Dermot Walsh
bmj.com, 11 Mar 2008 [Full text]

How does antidepressant medication relate to the pathophysiology of depression ?

Les O. Simpson
bmj.com, 11 Mar 2008 [Full text]

Questions remain

Paul R Matthews
bmj.com, 13 Mar 2008 [Full text]

limits of RCTs?

Declan P Fox
bmj.com, 12 Mar 2008 [Full text]

Neuropharmacological therapy of endogenous depression

Fuad Lechin, et al.
bmj.com, 13 Mar 2008 [Full text]

Correction of typographical error and competing interest addendum

Erick H. Turner
bmj.com, 13 Mar 2008 [Full text]

What about diminishing marginal returns?

Lucy J Robinson
bmj.com, 15 Mar 2008 [Full text]

Why GPs are indifferent to research based evidence. The case of antidepressant prescribing.

David Kernick
bmj.com, 17 Mar 2008 [Full text]

ERRATUM: What about diminishing marginal returns?

Lucy J Robinson
bmj.com, 19 Mar 2008 [Full text]

Use/misuse/abuse of antidepressants--a matter of serious concern

Tanu Pramanik.BSc,MA.(Sociology), et al.
bmj.com, 19 Mar 2008 [Full text]

Re: Why GPs are indifferent to research based evidence. The case of antidepressant prescribing.

P Ken Gillman
bmj.com, 19 Mar 2008 [Full text]

Depression, antidepressants, placebo effects and Peter Skrabanek.

Colin Brewer
bmj.com, 22 Mar 2008 [Full text]

Limitations in using sum-scores to assess antidepressant efficacy

Ulrik F Malt
bmj.com, 1 Apr 2008 [Full text]