Intended for healthcare professionals

CCBYNC Open access

Rapid response to:

Research

Efficacy of psilocybin for treating symptoms of depression: systematic review and meta-analysis

BMJ 2024; 385 doi: https://doi.org/10.1136/bmj-2023-078084 (Published 01 May 2024) Cite this as: BMJ 2024;385:e078084

Linked Editorial

Psilocybin for depression

Rapid Response:

Maintaining rational expectations about psychedelic interventions

Dear Editor,

We have serious concerns about the validity of the recently published meta-analysis that finds large benefits of psilocybin for symptoms of depression[1].

First, several Hedge’s g effect sizes (ES) in Figure 3 are larger than 3 and as high as 4.52 [2]. This is highly unlikely for psychiatric interventions and the original studies report much smaller ESs. Unfortunately, authors did not provide study-level aggregate information (e.g., mean, SD, N), leaving multiple reasons that could account for this discrepancy. One is the use of standard errors in place of standard deviations when calculating ES. Another is lack of transparency about how the main outcome measure (change from baseline) was calculated for trials that did not report the standard deviation of the change, where the pre-post correlation would be needed to compute ESs. Further, it is unclear how cross-over trials were handled when insufficient information for calculating within group differences was reported. All three issues can inflate ESs.

Second, there are reporting and analytical inconsistencies. The number of included trials differs between 7 (Figure 3 and summary tables) and 6 (Table 1: 3 parallel [3-5], 3 cross-over [2, 6, 7] and 3 secondary analyses [8-10] of the same primary trials). In one case, the largest trial (NCT03775200) appears to have been included twice in Figure 3, most likely with different outcomes (MADRS reported in the primary report [5], QIDS-SR-16 in the secondary analysis [9]). But another study [4] which also included two outcomes (MADRS and BDI) appears only once. Authors do not describe how they deal with dependency between ESs in cases the same sample of patients is analyzed multiple times, which requires the use of established methods [11]; not doing so can inflate ESs.

Third, risk of bias is evaluated as low for most domains across trials, at odds with a systematic review of psychedelic trials concluding that nearly all trials “were at least rated as high risk of bias in the domain ‘Measurement of the outcome’ partly due to unsuccessful blinding or lack of reporting of blinding of outcome assessor.” [12] Psychedelic trials may be designed and intended to be double-blind, but functional unblinding renders them effectively open-label. In the present study [1], for at least one of the included trials [4] the investigators themselves discuss functional unblinding and potential nocebo effects as potential explanations for the results. Further, all cross-over trials were rated as low risk for carryover effects, which is unlikely particularly when a high dose was given first. Moreover, all 3 cross-over trials [2, 6, 7] conclude that effects were sustained at 6 months, which is at odds with the lack of carryover effects.

In sum, ESs are likely inflated due to methodological flaws. This explains why the reported aggregate effect for the continuous depression outcomes (Hedge’s g=1.64) is beyond those observed for most medical interventions [13], and much better than established depression treatments such as psychotherapies[14] or antidepressants[15]. Of note, estimates also differ from another recently published (and uncited) meta-analysis on the same topic [16], which included more studies (9) and reported a pooled effect that, while still large, is more in line with estimates from treatments with early phase trials.

Unfortunately, it is not possible to inspect the dataset because it has not been made publicly available (concerns regarding participants’ privacy do not apply to such aggregated data [17]). Transparency is paramount particularly for publications as this meta-analysis, which could influence drug approval. We hope that our response will lead to authors posting the data; the next steps involve a careful inspection of the data, an independent re-analysis, and, likely, a correction of the record.

Ioana A. Cristea [1], Eiko I. Fried [2], Tim Kaiser [3], Michael P. Hengartner [4], Erick H. Turner [5], Florian Naudet [6,7]

1) Department of General Psychology, University of Padova, Padova, Italy
2) Clinical Psychology Unit, Institute of Psychology, Leiden University, The Netherlands
3) Methods and Evaluation / Quality Assurance Unit, Freie Universität Berlin, Berlin, Germany
4) Department of Applied Psychology, Zurich University of Applied Sciences, Switzerland
5) Department of Psychiatry, Oregon Health & Science University, Portland, Oregon USA
6) Univ Rennes, CHU Rennes, Inserm, EHESP, Irset (Institut de recherche en santé, environnement et travail) - UMR_S 1085, Centre d’investigation clinique de Rennes (CIC1414), F-35000 Rennes, France
7) Institut Universitaire de France, Paris, France.

REFERENCES
1. Metaxa A.M., Clarke M. 2024 Efficacy of psilocybin for treating symptoms of depression: systematic review and meta-analysis. Bmj 385, e078084. (doi:10.1136/bmj-2023-078084).
2. Griffiths R.R., Johnson M.W., Carducci M.A., Umbricht A., Richards W.A., Richards B.D., Cosimano M.P., Klinedinst M.A. 2016 Psilocybin produces substantial and sustained decreases in depression and anxiety in patients with life-threatening cancer: A randomized double-blind trial. J Psychopharmacol 30(12), 1181-1197. (doi:10.1177/0269881116675513).
3. Carhart-Harris R., Giribaldi B., Watts R., Baker-Jones M., Murphy-Beiner A., Murphy R., Martell J., Blemings A., Erritzoe D., Nutt D.J. 2021 Trial of Psilocybin versus Escitalopram for Depression. N Engl J Med 384(15), 1402-1411. (doi:10.1056/NEJMoa2032994).
4. von Rotz R., Schindowski E.M., Jungwirth J., Schuldt A., Rieser N.M., Zahoranszky K., Seifritz E., Nowak A., Nowak P., Jäncke L., et al. 2023 Single-dose psilocybin-assisted therapy in major depressive disorder: A placebo-controlled, double-blind, randomised clinical trial. EClinicalMedicine 56, 101809. (doi:10.1016/j.eclinm.2022.101809).
5. Goodwin G.M., Aaronson S.T., Alvarez O., Arden P.C., Baker A., Bennett J.C., Bird C., Blom R.E., Brennan C., Brusch D., et al. 2022 Single-Dose Psilocybin for a Treatment-Resistant Episode of Major Depression. N Engl J Med 387(18), 1637-1648. (doi:10.1056/NEJMoa2206443).
6. Grob C.S., Danforth A.L., Chopra G.S., Hagerty M., McKay C.R., Halberstadt A.L., Greer G.R. 2011 Pilot study of psilocybin treatment for anxiety in patients with advanced-stage cancer. Arch Gen Psychiatry 68(1), 71-78. (doi:10.1001/archgenpsychiatry.2010.116).
7. Ross S., Bossis A., Guss J., Agin-Liebes G., Malone T., Cohen B., Mennenga S.E., Belser A., Kalliontzi K., Babb J., et al. 2016 Rapid and sustained symptom reduction following psilocybin treatment for anxiety and depression in patients with life-threatening cancer: a randomized controlled trial. J Psychopharmacol 30(12), 1165-1180. (doi:10.1177/0269881116675512).
8. Barba T., Buehler S., Kettner H., Radu C., Cunha B.G., Nutt D.J., Erritzoe D., Roseman L., Carhart-Harris R. 2022 Effects of psilocybin versus escitalopram on rumination and thought suppression in depression. BJPsych Open 8(5), e163. (doi:10.1192/bjo.2022.565).
9. Goodwin G.M., Aaronson S.T., Alvarez O., Atli M., Bennett J.C., Croal M., DeBattista C., Dunlop B.W., Feifel D., Hellerstein D.J., et al. 2023 Single-dose psilocybin for a treatment-resistant episode of major depression: Impact on patient-reported depression severity, anxiety, function, and quality of life. J Affect Disord 327, 120-127. (doi:10.1016/j.jad.2023.01.108).
10. Ross S., Agin-Liebes G., Lo S., Zeifman R.J., Ghazal L., Benville J., Franco Corso S., Bjerre Real C., Guss J., Bossis A., et al. 2021 Acute and Sustained Reductions in Loss of Meaning and Suicidal Ideation Following Psilocybin-Assisted Psychotherapy for Psychiatric and Existential Distress in Life-Threatening Cancer. ACS Pharmacol Transl Sci 4(2), 553-562. (doi:10.1021/acsptsci.1c00020).
11. Rücker G., Cates C.J., Schwarzer G. 2017 Methods for including information from multi-arm trials in pairwise meta-analysis. Res Synth Methods 8(4), 392-403. (doi:10.1002/jrsm.1259).
12. Hovmand O.R., Poulsen E.D., Arnfred S., Storebø O.J. 2023 Risk of bias in randomized clinical trials on psychedelic medicine: A systematic review. J Psychopharmacol 37(7), 649-659. (doi:10.1177/02698811231180276).
13. Leucht S., Helfer B., Gartlehner G., Davis J.M. 2015 How effective are common medications: a perspective based on meta-analyses of major drugs. BMC Med 13, 253. (doi:10.1186/s12916-015-0494-1).
14. Cuijpers P., Quero S., Noma H., Ciharova M., Miguel C., Karyotaki E., Cipriani A., Cristea I.A., Furukawa T.A. 2021 Psychotherapies for depression: a network meta-analysis covering efficacy, acceptability and long-term outcomes of all main treatment types. World Psychiatry 20(2), 283-293. (doi:https://doi.org/10.1002/wps.20860).
15. Cipriani A., Furukawa T.A., Salanti G., Chaimani A., Atkinson L.Z., Ogawa Y., Leucht S., Ruhe H.G., Turner E.H., Higgins J.P.T., et al. 2018 Comparative efficacy and acceptability of 21 antidepressant drugs for the acute treatment of adults with major depressive disorder: a systematic review and network meta-analysis. Lancet 391(10128), 1357-1366. (doi:10.1016/s0140-6736(17)32802-7).
16. Haikazian S., Chen-Li D.C.J., Johnson D.E., Fancy F., Levinta A., Husain M.I., Mansur R.B., McIntyre R.S., Rosenblat J.D. 2023 Psilocybin-assisted therapy for depression: A systematic review and meta-analysis. Psychiatry Res 329, 115531. (doi:10.1016/j.psychres.2023.115531).
17. Cristea I.A., Naudet F., Caquelin L. 2022 Meta-research studies should improve and evaluate their own data sharing practices. J Clin Epidemiol 149, 183-189. (doi:10.1016/j.jclinepi.2022.05.007).

Competing interests: We have read and understood BMJ policy on declarations of interests and have the following interests to declare: FN received funding from the French National Research Agency (ANR-17-CE36-0010), the French ministry of health and the French ministry of research. He is a work-package leader in the OSIRIS project (Open Science to Increase Reproducibility in Science). The OSIRIS project has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No. 101094725. He is a work-package leader for the doctoral network MSCA-DN SHARE-CTD (HORIZON-MSCA-2022-DN-01 101120360), funded by the EU. The other authors declare no competing interests.

04 May 2024
Ioana A. Cristea
Associate Professor
Eiko I. Fried, Tim Kaiser, Michael P. Hengartner, Erick H. Turner, Florian Naudet
Department of General Psychology
University of Padova, Padova, Italy