Ghost busters: why even high ranking psychotherapy studies might be lousy

img_4358The last half year saw two high ranking psychotherapy studies in depression, published in the most prestigious journals of our profession. While in principle this is laudable, in these specific cases it is not doing any good as the studies fall short to prove what they are actually claiming and also, because they sack medical care for the sake of cost. Let’s have a closer look.

The first paper (, on the “COBRA” study, was published in Lancet (!!!) and compared „behavioral activation” delivered by „junior mental health workers“ (i.e. relatively untrained and, first and foremost, receiving relatively small wages) is just as effective as routine CBT delivered by trained psychologist (which is the gold standard psychotherapy treatment in depression). So depression treatment can be quite unspecific and cheap! Yay!


The most important drawback is the that the primary endpoint was set at twelve months. As the average length of a depressive episode is six to eight month without treatment, this makes no sense at all. Imagine that a study on two treatments aimed at relieving common cold would set its primary endpoint at 4 weeks. Would you buy this? As neither a survival plot (Kaplan-Meier-Plot) nor any other time course are shown, being suspicious is appropriate.

A further, unfortunately quite common, drawback is the lack of a sham psychotherapy group (i.e., this study is not placebo controlled). Given the year-long course, it may quite well have shown that it is as effective as the two other groups.

This is made even worse by the fact that 80% of participants in both groups received anti-depressant drug treatment. Likely, a ceiling effect is effective, further obscuring any effect of whatever psychotherapy is done.

This is not a non-inferiority trial. This is a failed trail.

Another study published in JAMA Psychiatry ( echoes the COBRA study, although changing the flavor. Here, psychodynamic (not behavioral activation) was compared against CBT in depression. I don’t want to nag about the underpowered sample for a non-inferiority trial (especially when looking at how many patients attended more than five sessions (116 in total). Again, we have an endpoint which is rather late (5 month) without any description of time courses, and again sham psychotherapy is lacking. Even worse, that average Hamilton score (HAM-D, likely HAM-D 17) was 21±6 points. This is quite low and barely reaches the border to moderate depression; the usual cut-off for study inclusion in pharma trials lies between 20 and 22. This means that many patients with mild depression were included, that usually are not the target population for depression studies. Any differences to placebo/sham are hard to demonstrate due to floor effects, especially when considering the low number of patients adhering to therapy and the measured effect size of 0.6 (Cohen’s d, corresponding to a medium effect size). Considering all this, watchful waiting, mere psychoeducation or having a beer every week or so would have had the same effect, namely, a reduction of five points on the HAM-D 17, as this is just the naturalistic course of mild to medium depression. An indicator of this are the significantly overlapping SD measurements pre- and post-treatment (wisely enough, the authors did not go for graphical display of their data). The most parsimonious interpretation of the data thus is that both treatments are equally ineffective!

You may ask about antidepressant use here as well. There is a simple answer: we don’t know. The numbers of patients on antidepressants are not given at all! That they were there, we know, as there is a small subclause: „We found no statistically significant interaction between the use of psychotropic medication and treatment group on the rate of change in the HAM-D”.

This is another failed trail. Although it was highly published…

Unsurprisingly, the rate of recidivism (which is a major effect of psychotherapy) is not given in any of the studies.

It is very surprising however that these studies were published so well, despite of these obvious flaws. I can only speculate on the reasons for this. Regarding the Lancet paper (COBRA), I assume that economic reasons play a major role. Cheap BA treatment by “juniors” (did not we just learn to abandon that word from our vocabulary?) is as effective as CBT by expensive, greedy psychologist. That makes treatment less expensive, which however is somewhat tainted by the fact that it is ineffective (notwithstanding the commonsense experience that unspecific BA especially in early stage depression by be quite helpful). Never mind. The psychodynamic study might have undergone a “wishful thinking” review process – so many people are out there who desperately wish that psychodynamic therapy works as well as CBT… so this one came in quite handy. However, no favor was done to the field, on contrary. We do not need so badly designed (or at least presented) studies; what we do need are psychotherapy trials adhering to the highest standards in analogy to drug trials: i.e. presenting time courses, studying severe cases, being well powered with low attrition rates, and – most important – including a sham (=placebo) condition.

Transitioning from child to adult: a new declaration demands continuing support for ADHD in Europe

image ‘growing up’ by sillysirry via Diviantart

ADHD Europe – the European patient organisation for people with ADHD – has launched a declaration on behalf of teenagers with ADHD. The declaration states that adult services in all European countries should offer suitable care for teenagers with ADHD who transition into adulthood, and to adults who are newly diagnosed. Andrea Bilbow is the president of the ADHD Europe organisation. I asked her about the importance of this declaration, and what she thinks that the consequences of the declaration will be.

Why was this declaration to urgently needed?

“Well it’s a bit of a long story that goes back 20 years. We started as organisations to raise awareness for ADHD in children, and to organise good services for these children. In most European countries there now are good services for children with ADHD.”

“However, these children are all becoming adults. For many young people across Europe as soon as they reach 18 years of age, they find that transition to adult services is very poor.  In many cases treatment is withdrawn altogether leaving them vulnerable and at risk.  And for those fortunate enough to receive the medication that they need for ADHD in adulthood, the medication is no longer reimbursed. This puts a huge burden on families who often have two or more children with ADHD and who do not have the resources to pay for the medication. Only in one or two countries in Europe are there official adult licences for the medication needed by adults with ADHD. Besides medication, young adults with ADHD do not have access to the services for ADHD in adult clinics, while they were receiving help in children’s services.”

Why is it so important that adults with ADHD receive suitable care?

“18 is a very vulnerable age. It’s the age when adolescents move up to Higher Education or start with their first jobs. Having ADHD, these young people are vulnerable to substance abuse, academic failure, job loss, becoming homeless or even crime. Furthermore, many of them will be starting to drive. Studies have shown that  a significant number of young people with untreated ADHD will be involved in car accidents.”

“For years the EU has been trying to address the problem of school dropout in Europe, without success. Having medication for ADHD continue to be reimbursable and proper services for young adults with ADHD available would go a long way to reduce the number of young people who abandon education. So it is very, very important that they receive proper treatment and support. It’s probably one of the most important things you can do for this age group.”

People can now sign the declaration to show their support. What do you think will happen next?

“Well first of all, it’s amazing to see how much support we are receiving. The Declaration has been launched for only a week and already we have more than 500 signatures. More importantly, these signatures come from all over the world. But it also shows that what this declaration states is very much needed. Clearly, 18+ people across Europe are really struggling.”

“First, we will let this run for a couple of months to see how many signatures we can receive. We encourage professionals to also show their support: professors, doctors, medics, researchers, teachers, police. Anybody who has any stake in improving the lives of people with mental health problems. Once we have gathered this support, we will encourage the member countries to take it to their MEPs and try to get the required number of MEPs to sign the declaration in order for it to be discussed in the European Parliament. That is our mission. We will try to find out why  in some countries they don’t want psychiatrists to diagnose and treat adults with ADHD. This is a human right that’s being breached.”

This October is ADHD awareness month. The perfect moment for the ADHD Europe patient organisation to bring out this Declaration and to raise awareness for this important issue. You can find, and sign, the Declaration at the website of ADHD Europe:

The Declaration also shows the importance of research on ADHD. The CoCA project for example will investigate the societal costs of ADHD and comorbid disorders. Such data can assist in persuading governments about the importance of providing suitable care for those – children and adults- with ADHD.

This post was written by Jeanette Mostert. Jeanett is Dissemination Manager of the CoCA project.

Going bananas about methylphenidate studies!

Why did I chose to use the Minions as a feature image for this post, along with the catchy title? Simply to attract attention. Sheer clickbait. While this is perfectly acceptable for a blog post (well, almost…), it is not for scientific publications. This not only refers to the title of a paper, but also to the way it is disseminated; and in this respect, a series of manuscripts under the lead authorship from O. Storebo raised some brows with their bold claim that there is no evidence that methlyphenidate actually works. While most of us clinicians would readily agree that this medication requires experience, thorough assessment, responsibility, and that it is not rarely ill-prescribed (often however by doctors other than psychiatrists), most of us are sure that it is indeed an effective medication given that the ADHD diagnosis is valid. So are we all deluded?

Well… probably not. At least not when it comes to methylphenidate treatment.

The group around Storebo, who before worked in the ADHD research in one trial on social skills training (the SOSTRA study), conducted a Cochrane on the efficacy of methylphenidate in ADHD and found out that “methylphenidate may improve teacher-reported ADHD symptoms”, but “due to the very low quality of the evidence, the magnitude of the associated improvement is uncertain”. This led to some far-fetched conclusions and statements (see e.g. the conclusions section in the abstract here: and it was wide-spread communicated in the media that “methylphenidate is not effective”. A deleterious statement, which also outraged many patients and parents.

So far, so bad; Cochrane reviews are known for their methodological rigor, however there are many back doors so one should always look at them critically, as you can tweak your input. What was the fine-tuning done here?  To start with, the effect size estimate is based upon 19 (from 185 included) studies. 4 did investigate methylphenidate versus an active control, another study  was undertaken in children under 6 years (off-label). When these studies are excluded, which should have been done, effect sizes increase to a large effect of 0.89. On the other hand, 56 studies that employed a cross-over design were excluded for no clear and good reason.


This should be enough to cast doubt on the study. However, in addition, there was an unusually strict and almost arbitrary assessment of bias; this led the authors to rate ALL (!!!) 185 studies to be at high risk of bias – and hence categorizing all studies as “low quality”. However, the evidence to support this claim is little. Personally, I find it outragous (and as I did not take part in any of these studies, I am not biased…) especially as the most common source of potential bias were assumed “conflicts of interest”.  While I consider Disclosure of Interest as a very important thing, one cannot make a general accusation and suspect almost a whole speciality of being bribed. This is demagogue, not science. This categorization results in a striking devaluation of decades of evidence from RCTs and also contradicts e.g. a NICE review (

Finally, Storebo’s proposal to implement long-term nocebo-controlled studies – despite the strong actual evidence from several decades of RCTs on methylphenidate – implies to administer a substance with no known benefit, but significant side effects for a substantial time period to many patients including minors. In my opinion, this is deeply unethical and conflicts with §33 of the Declaration of Helsinki.

While the grounds for their bottomline claim may be slippery, the authors do a good job in selling it. They have published the original Cochrane review, followed by a publication of the very same data in the prestigious BMJ and another publication of the same dataset in JAMA Bear with me, but haven’t I been told in grad school that one of the Ten Commandments in Science is “Thou shalt not publish the same data twice”?

Just by repeating their interpretation over and over in high impact journals, the notion that methylphenidate is not working will trickle in the general conscience while the empirical basis for this claim suggests otherwise. This will impose harm on our patients, and this is why we have to address and disprove these papers actively.