Systematic reviews of health interventions should consider both harms and potential benefits
Mutz et al. investigated the efficacy of non-surgical brain stimulation such as electro-convulsive therapy (ECT) and repetitive transcranial magnetic stimulation (rTMS) for the acute treatment of adult depression but did not assess any harms . In addition to their efficacy outcomes, response, remission and a standardized mean change on depression symptom scales, they investigated total dropout rates as a measure of “acceptability”. The Cochrane Handbook advises that review authors hesitate to interpret such data as surrogate markers for harms or tolerability because of the potential for bias that can arise due to several issues . First, the attribution of reasons for discontinuation from a trial is complex and may be due to mild but irritating side effects, toxicity, lack of efficacy, non-medical reasons, or a combination of causes . Second, there may be pressures on participants and investigators involved in a trial to minimise the number of drop-outs, which can result in drop-out rates that do not reflect the true experience of adverse events among trial participants. Third, the decision to withdraw from a trial is often preceded by unblinding of intervention allocation and can distort the perception of the intervention’s effect on participant withdrawal. Lastly, if a similar number of participants withdraw from the intervention and control group, may go unrecognised if the reason for withdrawal in the control group is primarily lack of effect but primarily harmful effects in the intervention group.
Harms are often inadequately reported while efficacy results are emphasized or overstated in the reporting of randomised trials [4, 5]. Distorted reporting can potentially lead to a biased assessment of interventions and to a misleading conception of absence of harms when there is actually uncertainty . Systematic reviews often compound poor reporting of harms in the published literature by focusing on efficacy while failing to report harms or doing so inadequately . Recent efforts have sought to improve the harms reporting in systematic reviews, such as the development of the PRISMA harms checklist, endorsed by The BMJ .
Investigators performing systematic reviews that assess harms often face challenges, including finding and selecting data, rating the quality of harms reporting, and synthesizing and displaying data from individual studies . It is unclear from the article by Mutz et al. whether any of these concerns underlie their lack of harms outcomes and there is no protocol available for the review, which may have provided such information . Acknowledging that they did not “examine specific undesired and adverse effects”, Mutz et al. state that they will do so in a future review . However, the cited published protocol for that review concerns plans to review “cognitive effects” of non-surgical brain stimulation only, but no harms beyond that . While there are known cognitive harms of ECT, which, in some cases, may be permanent , these are not the only potential harms of non-surgical brain stimulation. The harms of rTMS, among others, include seizures and hypomania . Known harms of ECT include death, physical trauma, prolonged seizures and cardiovascular complications . Any assessment of the clinical utility of these interventions must necessarily include an assessment of these and other harms.
It is a clearly stipulated requirement that in order to provide useful guidance on an intervention, evidence syntheses should assess the balance of critical benefits and harms, including an assessment of the certainty of these outcomes . Mutz et al., however, did neither evaluate the certainty of the evidence for the efficacy of non-surgical brain stimulation nor did they evaluate any harms. Therefore, the systematic review by Mutz et al., in contrast to their anticipation, cannot have implications for clinical decision making. Further, by focusing only on potential benefits and disregarding harms the review potentially distorts the assessment of non-surgical brain stimulation interventions for depression.
1. Mutz J, Vipulananthan V, Carter B, et al. Comparative efficacy and acceptability of non-surgical brain stimulation for the acute treatment of major depressive episodes in adults: systematic review and network meta-analysis. BMJ 2019;364:l1079. doi: 10.1136/bmj.l1079 [published Online First: 2019/03/29]
2. Higgins JPT, Green S, Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions. Version 5.1.0 [updated March 2011]. Available from www.cochrane-handbook.org.2008.
3. Ioannidis JP, Evans SJ, Gotzsche PC, et al. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med 2004;141(10):781-8. [published Online First: 2004/11/17]
4. Pitrou I, Boutron I, Ahmad N, et al. Reporting of safety results in published reports of randomized controlled trials. Archives of internal medicine 2009;169(19):1756-61. doi: 10.1001/archinternmed.2009.306 [published Online First: 2009/10/28]
5. Ioannidis JP, Lau J. Completeness of safety reporting in randomized trials: an evaluation of 7 medical areas. JAMA 2001;285(4):437-43. [published Online First: 2001/03/10]
6. Ioannidis JP. Adverse events in randomized trials: neglected, restricted, distorted, and silenced. Archives of internal medicine 2009;169(19):1737-9. doi: 10.1001/archinternmed.2009.313 [published Online First: 2009/10/28]
7. Zorzela L, Loke YK, Ioannidis JP, et al. PRISMA harms checklist: improving harms reporting in systematic reviews. BMJ 2016;352:i157. doi: 10.1136/bmj.i157 [published Online First: 2016/02/03]
8. Chou R, Helfand M. Challenges in systematic reviews that assess treatment harms. Ann Intern Med 2005;142(12 Pt 2):1090-9. [published Online First: 2005/06/22]
9. Kiebs M, Hurlemann R, Mutz J. Cognitive effects of non-surgical brain stimulation for major depressive disorder: protocol for a systematic review and meta-analysis. BMJ Open 2019;9(2):e023796. doi: 10.1136/bmjopen-2018-023796 [published Online First: 2019/02/21]
10. Association AP. The Practice of Electroconvulsive Therapy: Recommendations for Treatment, Training, and Privileging (A Task Force Report of the American Psychiatric Association): American Psychiatric Association 2008.
11. Loo CK, McFarquhar TF, Mitchell PB. A review of the safety of repetitive transcranial magnetic stimulation as a clinical treatment for depression. Int J Neuropsychopharmacol 2008;11(1):131-47. doi: 10.1017/S1461145707007717 [published Online First: 2007/09/21]
12. regulations.gov. Neurological Devices; Reclassification of Electroconvulsive Therapy Devices Intended for Use in Treating Severe Major Depressive Episode in Patients 18 Years of Age and Older Who Are Treatment Resistant or Require a Rapid Response; Effective Date of Requirement for Premarket Approval for Electroconvulsive Therapy for Certain Specified Intended Uses, 2015.
13. Atkins D, Best D, Briss PA, et al. Grading quality of evidence and strength of recommendations. BMJ 2004;328(7454):1490. doi: 10.1136/bmj.328.7454.1490 [published Online First: 2004/06/19]
Competing interests: No competing interests