Are Methylphenidate Effects in Children with ADHD Really Uncertain?
Recently Storebø et al. published in the BMJ  a summary of their Cochrane review  on the efficacy and tolerability of methylphenidate (MPH) for the treatment of attention-deficit/hyperactivity disorder (ADHD) in children and adolescents. In recent years, numerous systematic reviews have consistently demonstrated the high degree of efficacy of MPH. MPH is currently recommended as a first-line treatment for moderate to severe ADHD by the NICE.  While the present review reports similar effect sizes in relation to efficacy, it suggests that the findings may be inaccurate because the quality of the evidence is weak due to high levels of bias in all of the included studies. The conclusions of Storebø et al. could lead readers to question the value of MPH in the treatment of ADHD. However, there are a number of errors and biases in the Storebø et al. meta-analysis and its interpretation. We highlight here the most important ones.
Errors in study inclusion. The authors aimed to include trials “comparing all types of methylphenidate with placebo or no intervention.” Given this, three studies were incorrectly included, where there was no placebo/no treatment arm [4, 5] and/or where MPH was used as an add-on intervention.[4-6] Both designs are likely to under-estimate the effect of MPH. Removing these studies increases the effect size for the primary outcome of teacher-reported ADHD symptom ratings from -0.77 (95% CIs -0.90, -0.64) to -0.83 (95% CIs -0.96; - 0.70). One trial incorrectly included with no placebo/no treatment arm was the large MTA study.  This affects all the subgroup analyses of long-term (>6 months) versus short-term MPH administration. Storebø et al. report a significantly smaller effect for long-term administration according to teacher (but not parent or observer) reports and argue this casts doubt on the long-term benefits of MPH. However, MTA is the only study meeting the authors’ definition of long-term administration; therefore, these analyses should not have been undertaken and the results should be ignored.
Assessment of study quality. Storebø et al. adopted the GRADE approach to the assessment of quality, which includes the Cochrane risk of bias (RoB) tool. Using the GRADE approach, with regards to the primary outcome (teacher-rated ADHD symptoms), they downgraded the quality of evidence by one point for inconsistency of effects (heterogeneity) and by two points for risk of bias. Both these decisions are questionable. In relation to heterogeneity, I-squared for the meta-analysis of the main outcome was 37% for the primary outcome measure. The Cochrane Handbook suggests that heterogeneity up to 40% may not adversely affect the findings.  Of note, heterogeneity was increased by the inappropriate inclusion of MTA study; we calculated that heterogeneity drops to 25% when this study is removed.
With regard to the RoB assessment, we identified two major deviations from current RoB standards. First, Storebø et al. added an additional domain, ‘vested interests’, which is not included in the current RoB tool. This domain was poorly defined and inconsistently rated. Furthermore, from reviewing the tables presented in the Cochrane, there is no evidence that Storebø et al. had routinely attempted to contact authors for clarification when this domain was rated as ‘unclear’. This seemed strange as they had attempted to obtain missing quantitative data on trial outcomes. In a BMJ online reply to the concerns of others about their use of the vested interest domain, the authors stated “There were no trials with only the 'vested interest bias' domain assessed as 'unclear risk of bias' or 'high risk of bias.'” This is incorrect as there are actually seven studies in which this is the only domain of risk identified.
Second, Storebø et al. rated individual studies as having a high risk of bias if at least one domain (including their additional domain of vested interests) was rated as high or unclear. This approach is problematic because some domains were rated as unclear simply due to lack of detailed information that was not specifically and systematically sought from the authors. Storebø et al. argue that these deficits in reporting, which led to idiosyncratic ratings of ‘high risk of bias’ in almost all studies, inevitably cast doubt on the assessments of effect sizes and they argue these should therefore be treated with great caution. The authors themselves undertook subgroup analyses, presented in the fuller Cochrane review , comparing RCTS at high versus low risk of all bias (their ratings) and failed to find a significant difference in effect size (χ-squared, 1 df=2.43, p=.12). Surprisingly, given their concern about bias, they failed to include this important finding in their BMJ paper.
Serious adverse effects of MPH. Storebø et al. state that “methylphenidate has been reported to cause rare but sudden cardiac death” (p 10, BMJ) but the article they cite in fact concludes “Growing evidence suggests that these medications (referring to stimulants) do not cause sudden and unexpected cardiac death or serious cardiovascular problems… at therapeutic doses in ADHD patients across the lifespan”. 
Errors in computation of effect size. We reviewed the studies included in Storebø et al.’s primary analysis of teacher-reported ADHD symptoms. Among the 19 studies reported (three of which should not have been included), we found errors in the imputation of data and/or sample size in seven (44% of the 16 eligible studies), including a systematic doubling of the sample sizes for all crossover trials (by counting each arm as independent). Our limited check raises serious concerns about the other quantitative findings (which could be substantially affected in terms of effect size and confidence interval calculation) and the overall quality and conduct of the review.
In summary, we believe that the review by Storebø et al. is seriously flawed in its methods, data collection and interpretation of findings. Its conclusions should neither alter current clinical practice nor affect the confidence of practitioners, parents/carers and patients in the efficacy of MPH treatment for ADHD.
Authors & Affiliates:
Tobias Banaschewski (a),
Jan Buitelaar (b),
Celine S. L. Chui (c),
David Coghill (d),
Samuele Cortese (e),
Emily Simonoff (f)*,
Ian C. K. Wong (g)
On behalf of the European ADHD Guidelines Group: Philip Asherson (h), Daniel Brandeis (i), David Daley (j), Ralf W. Dittmann (k), Marina Danckaerts (l), Manfred Doepfner (m), Maite Ferrin (n), Chris Hollis (j), Martin Holtmann (o), Eric Konofal (p), Michel Lecendreux (q), Aribert Rothenberger (q), Paramala Santosh (r), Edmund Sonuga-Barke (e), César Southillo (s), Hans-Christoph Steinhausen (t); (u), Argyris Stringaris (f), Eric Taylor (f), Saskia Van der Oord (v), Alessandro Zuddas (w).
(a). Department of Child Adolescent Psychiatry and Psychotherapy, Central Institute of Mental Health, Mannheim, Germany
(b). Department of Cognitive Neuroscience, Donders Institute of Brain, Cognition and Behavior, Radboud University Medical Center, Nijmegen, The Netherlands, and Karakter Child and Adolescent Psychiatry University Center, Nijmegen, The Netherlands
(c). Department of Pharmacology and Pharmacy, University of Hong Kong, Hong Kong, China
(d). Departments of Paediatrics and Psychiatry, University of Melbourne, Victoria, Australia
(e). Department of Psychology, University of Southampton, Southampton, UK and Clinical and Experimental Sciences (CNS and Psychiatry), Faculty of Medicine, University of Southampton
(f). Department of Child & Adolescent Psychiatry, King’s College London, Institute of Psychiatry Psychology & Neuroscience, London, UK*
(g). UCL School of Pharmacy, Institute of Child Health, London, UK
(h). Social Genetic and Developmental Psychiatry, King’s College London, Institute of Psychiatry Psychology & Neuroscience, London, UK
(i). Department of Child & Adolescent Psychiatry and Psychotherapy, Psychiatric Hospital, University of Zurich, Switzerland and Department of Child & Adolescent Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
(j). Division of Psychiatry and Applied Psychology, Institute of Mental Health, School of Medicine University of Nottingham, Nottingham, UK
(k). Department of Child & Adolescent Psychiatry, Central Institute of Mental Health, University of Heidelberg, Mannheim, Germany
(l). Child & Adolescent Psychiatry, University Psychiatric Centre KU Leuven, Leuven, Belgium
(m). Department of Child & Adolescent Psychiatry and Psychotherapy, Medical Faculty, University of Cologne, Cologne, Germany
(n). Developmental Brain and Behaviour Laboratory, University of Southampton, Southampton, UK
(o). Department of Child & Adolescent Psychiatry, Psychotherapy and Psychosomatic Medicine, Ruhr-University Bochum, Hamm, Germany
(p). Pediatric Sleep Center, Hospital Robert Debré, Paris, France
(q). Clinic for Child and Adolescent Psychiatry, Goettingen, Germany
(r). Department of Child & Adolescent Psychiatry, Centre for Interventional Paediatric Psychopharmacology and Rare Diseases (CIPPRD), King’s College London, UK
(s). Child & Adolescent Psychiatry Unit, Department of Psychiatry & Medical Psychology, University of Navarra Clinic, Pamplona, Spain
(t). Department of Child and Adolescent Psychiatry, University of Zurich, Switzerland
(u). Clinical Psychology and Epidemiology, University of Basel, Switzerland
(v). Department of Psychology and Educational Sciences, KU Leuven, Leuven, Belgium
(w). Department of Biomedical Science, University of Cagliari, Cagliari, Italy
*Correspondence to: firstname.lastname@example.org
We have read and understood the BMJ Group policy on declaration of interests and declare the following interests:
The Corresponding Author has the right to grant on behalf of all authors and does grant on behalf of all authors, a worldwide licence to the Publishers and its licensees in perpetuity, in all forms, formats and media (whether known now or created in the future), to i) publish, reproduce, distribute, display and store the Contribution, ii) translate the Contribution into other languages, create adaptations, reprints, include within collections and create summaries, extracts and/or, abstracts of the Contribution, iii) create any other derivative work(s) based on the Contribution, iv) to exploit all subsidiary rights in the Contribution, v) the inclusion of electronic links from the Contribution to third party material where-ever it may be located; and, vi) licence any third party to do any or all of the above.
All authors certify that they have made:
- Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND
- Drafting the work or revising it critically for important intellectual content; AND
- Final approval of the version to be published; AND
- Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved
1. Storebø OJ, Krogh HB, Ramstad E, et al. Methylphenidate for attention-deficit/hyperactivity disorder in children and adolescents: Cochrane systematic review with meta-analyses and trial sequential analyses of randomised clinical trials. British Medical Journal 2015;351.
2. Storebø OJ, Ramstad E, Krogh HB, et al. Methylphenidate for children and adolescents with attention deficit hyperactivity disorder (ADHD). Cochrane Database of Systematic Reviews 2015(12).
3. National Collaborating Centre for Mental Health. Attention deficit hyperactivity disorder: Diagnosis and management of ADHD in children, young people and adults. In: Excellence NIfHaC, ed. London, 2008.
4. Firestone P, Kelly MJ, Goodman JT, et al. Differential effects of parent training and stimulant medication with hyperactives: A progress report. J Am Acad Child Psychiatry 1981;20(1):135-47.
5. The MTA Cooperative Group. A 14-month randomized clinical trial of treatment strategies for attention-deficit hyperactivity disorder. Archives of General Psychiatry 1999;56:1073-86.
6. Brown RT, Wynne ME, Medenis R. Methylphenidate and cognitive therapy: a comparison of treatment approaches with hyperactive boys. Journal of Abnormal Child Psychology 1985;13(1):69-87.
7. Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions version 5.1.0: The Cochrane Collaboration, 2011.
8. Martinez-Raga J, Knecht C, Szerman N, et al. Risk of serious cardiovascular problems with medications for attention-deficit hyperactivity disorder. CNS Drugs 2013;27(1):15-30.
This response was originally posted on 22 July but was re-posted on 28 July with slight modifications at the authors' request--Sharon Davies, The BMJ
Competing interests: No competing interests