Why did I chose to use the Minions as a feature image for this post, along with the catchy title? Simply to attract attention. Sheer clickbait. While this is perfectly acceptable for a blog post (well, almost…), it is not for scientific publications. This not only refers to the title of a paper, but also to the way it is disseminated; and in this respect, a series of manuscripts under the lead authorship from O. Storebo raised some brows with their bold claim that there is no evidence that methlyphenidate actually works. While most of us clinicians would readily agree that this medication requires experience, thorough assessment, responsibility, and that it is not rarely ill-prescribed (often however by doctors other than psychiatrists), most of us are sure that it is indeed an effective medication given that the ADHD diagnosis is valid. So are we all deluded?
Well… probably not. At least not when it comes to methylphenidate treatment.
The group around Storebo, who before worked in the ADHD research in one trial on social skills training (the SOSTRA study), conducted a Cochrane on the efficacy of methylphenidate in ADHD and found out that “methylphenidate may improve teacher-reported ADHD symptoms”, but “due to the very low quality of the evidence, the magnitude of the associated improvement is uncertain”. This led to some far-fetched conclusions and statements (see e.g. the conclusions section in the abstract here: http://www.ncbi.nlm.nih.gov/pubmed/26599576) and it was wide-spread communicated in the media that “methylphenidate is not effective”. A deleterious statement, which also outraged many patients and parents.
So far, so bad; Cochrane reviews are known for their methodological rigor, however there are many back doors so one should always look at them critically, as you can tweak your input. What was the fine-tuning done here? To start with, the effect size estimate is based upon 19 (from 185 included) studies. 4 did investigate methylphenidate versus an active control, another study was undertaken in children under 6 years (off-label). When these studies are excluded, which should have been done, effect sizes increase to a large effect of 0.89. On the other hand, 56 studies that employed a cross-over design were excluded for no clear and good reason.
This should be enough to cast doubt on the study. However, in addition, there was an unusually strict and almost arbitrary assessment of bias; this led the authors to rate ALL (!!!) 185 studies to be at high risk of bias – and hence categorizing all studies as “low quality”. However, the evidence to support this claim is little. Personally, I find it outragous (and as I did not take part in any of these studies, I am not biased…) especially as the most common source of potential bias were assumed “conflicts of interest”. While I consider Disclosure of Interest as a very important thing, one cannot make a general accusation and suspect almost a whole speciality of being bribed. This is demagogue, not science. This categorization results in a striking devaluation of decades of evidence from RCTs and also contradicts e.g. a NICE review (http://www.ncbi.nlm.nih.gov/pubmed/16796929).
Finally, Storebo’s proposal to implement long-term nocebo-controlled studies – despite the strong actual evidence from several decades of RCTs on methylphenidate – implies to administer a substance with no known benefit, but significant side effects for a substantial time period to many patients including minors. In my opinion, this is deeply unethical and conflicts with §33 of the Declaration of Helsinki.
While the grounds for their bottomline claim may be slippery, the authors do a good job in selling it. They have published the original Cochrane review http://www.ncbi.nlm.nih.gov/pubmed/26599576, followed by a publication of the very same data in the prestigious BMJ http://www.ncbi.nlm.nih.gov/pubmed/26608309 and another publication of the same dataset in JAMA http://www.ncbi.nlm.nih.gov/pubmed/27163989. Bear with me, but haven’t I been told in grad school that one of the Ten Commandments in Science is “Thou shalt not publish the same data twice”?
Just by repeating their interpretation over and over in high impact journals, the notion that methylphenidate is not working will trickle in the general conscience while the empirical basis for this claim suggests otherwise. This will impose harm on our patients, and this is why we have to address and disprove these papers actively.
Mind the Gap Blog
Fully agree,
Best wishes
Tobias Banaschewski
see also:
Banaschewski, T., et al. (2016). “Trust, but verify. The errors and misinterpretations in the Cochrane analysis by O. J. Storebo and colleagues on the efficacy and safety of methylphenidate for the treatment of children and adolescents with ADHD.” Z Kinder Jugendpsychiatr Psychother 44(4): 307-314.
OBJECTIVE: A recent Cochrane review published by O. J. Storebo and colleagues (2015) raised substantial doubts about the benefit from stimulant medication with methylphenidate in the treatment of childhood ADHD due to the overall poor quality of studies. The systematic review thus contradicts all previous reviews and meta-analyses. METHOD: We here detail various examples of errors, inconsistencies, and misinterpretations in the review which led to false results and inadequate conclusions. RESULTS: We demonstrate that the study selection is flawed and undertaken without sufficient scientific justification resulting in an underestimation of effect sizes, which, furthermore, are inadmissibly clinically interpreted. The methodology of the assessment of bias and quality is not objective and cannot be substantiated by the data. CONCLUSIONS: Cochrane reviews lay claim to a high scientific quality and substantial relevance for evidence-based clinical decisions. The systematic review by Storebo and colleagues (2015) illustrates that, despite adhering to strict standards and high-quality protocols, even Cochrane works should be critically read and verified, sometimes with surprising results.
LikeLike