Rapid Responses to:

LETTERS: Tom Sensky, Jan Scott, Brian Darnley, Paul Blenkiron, and Alison Tonks All you need is cognitive behaviour therapy? BMJ 2002; 324: 1522 [Full text]

Rapid Responses: Submit a response to this article

Rapid Responses published:

Repeated claims for the benefits of CBT do not strengthen the weak evidence

Nick Bolsover   (2 October 2002)

Repeated claims for the benefits of CBT do not strengthen the weak evidence 2 October 2002
Nick Bolsover, Consultant Clinical Psychologist in Psychotherapy Hull and East Riding Community Healthcare NHS Trus Miranda House, Gladstone Street, Hull, HU3 2RT Send response to journal: Re: Repeated claims for the benefits of CBT do not strengthen the weak evidence	Sensky and Scott1 suggest that my questioning of the CBT literature uses arguments which are instances of “idiosyncratic use of the research evidence” and calls for “clinicians and commentators to understand and respect the critical appraisal of the evidence base”. This turns my argument on its head as I was suggesting that it is CBT researchers who show a lack of respect for their chosen method, including presenting idiosyncratic versions of randomised control trials (RCTs) and being creative with their use of statistical analysis. Sensky and Scott remind me of the anacronym2 effect - they seem to think that if they give the impression that there is unequivocal evidence for the effectiveness of CBT this will make it so. However, like Garfield3 I find that “just a cursory glance” at the literature leads to questioning of the claims made for validated therapies/evidence based psychotherapy. There was no need for me to be selective about the papers I cited as claiming more than their data suggest. Papers I have quoted previously and will go on to quote below either immediately come to hand or are those suggested by reviews of CBT and rated as good evidence for CBT. For example, Leff et al4 is worth returning to as it illustrates many of the problems in psychotherapy RCTs as well as being a study in which CBT, although marketed as user friendly, was unable to engage participants in its arm of the study. This trial is described as an RCT and reached the conclusion that “couple therapy is much more acceptable than anti depressant drugs and is at least as efficacious, if not more so, both in treatment and maintenance phases”. In this study there was not a no treatment control group; two outcome measures were used but the p values were not adjusted; only one of the two outcome measures rejected the null hypothesis, at the five percent level; and, furthermore, the outcome measure that gave the significant effect did not have a significant group x occasion interaction. Therefore, this study is not strictly speaking a randomised control trial (it does not have a no treatment condition) and it does not strictly conform to the principles of statistical analysis. Two outcome measures are used and the p value should be halved as a consequence. This would mean that the main effect could not be reported as statistically significant. Furthermore, as there is not a significant occasion x treatment condition interaction it is not possible to state that the observed difference between treatment groups is a consequence of the treatment interventions. Since I started looking more closely at the evidence based literature, rather than accepting authors’ assertions, I have been surprised to see how frequently those who advocate validation of treatments via RCTs stretch the definition of RCT and are creative in their use of statistics. I have looked, as Sensky and Scott challenge me to, at the abstracts of the Cochrane Reviews5 of cognitive therapy. I noted the Cochrane disclaimer – the reviews are open to different interpretations. While Sensky and Scott obviously feel the evidential pot is half full, I see it as half empty. I particularly noted the Cochrane review of Cognitive Therapy in Schizophrenia6 and did not find the ringing endorsement their promotion of CBT led me to expect – “Reviewers' conclusions: Cognitive behavioural therapy is a promising but under evaluated intervention. Currently, trial-based data supporting the wide use of cognitive behavioural therapy for people with schizophrenia or other psychotic illnesses are far from conclusive. More trials are justified, especially in comparison with a lower grade supportive approach. These trials should be designed to be both clinically meaningful and widely applicable.” These conclusions remind me of earlier examples of the literature, in particular the British Psychological Society’s review of psychological approaches to psychotic experiences7. This included an assertion of the effectiveness of CBT, although examination of the papers cited revealed less papers than expected (each time a reference was mentioned it was given a new number – creating the impression that more papers were being cited than was actually the case) and much weaker evidence than readers were led to anticipate. That is, in section “12.3.4 The effectiveness of CBT” 12 references were cited, “Several published reports of randomised controlled trials (the ‘gold standard ‘ of clinical research) are available” - but only two were cited. The 12 references were of two RCTs, three treatment manuals, a review, a drug study and a controlled trial. Three papers were quoted twice and two of those (ie four of the 12 references) were, in fact, different parts of the same study. The “RCT”, which presented the more robust data, used multiple outcome measures and did not include a no treatment control group. The other RCT used numerous outcome measures, only one of which distinguished between groups, did not include a no treatment control and did not assess outcomes blind. That is, although readers were given to understand that the effectiveness of CBT is based on ‘several….randomised control trials’ actually 2 RCTs were referenced and neither were, strictly speaking, a RCT. That review7 stated, as Sensky and Scott seem to agree, “There is convincing evidence that psychological interventions (ie CBT), are effective for many people in reducing psychotic experiences and the distress and disability they cause”. I think this overestimates the evidence (as indicated by a Cochrane Review6), particularly when the evidence for other forms of psychotherapy is not even considered to deserve a mention. This claim is based on ‘gold standard’ studies which are not RCTs and the outcomes of which are not as clear cut as is suggested. For example, the first ‘RCT’ I refer to above, a study reported by Sensky and colleagues8, showed no difference between cognitive therapy (CT) and the befriending control at the end of the treatment phase, only at follow up did a difference emerge. The latter study also reports multiple outcome measures for which the p values have not been adjusted. Had the p values been appropriately adjusted it is unlikely that the authors could have reported a positive outcome at follow -up. Furthermore, this study reports the percentage of patients “who showed 50% or greater reduction in outcome scores at follow-up examination” as if supporting the use of CT, despite the analyses finding a statistically significant advantage for CT on only one of four measures (had the p values been adjusted for multiple outcomes none of the measures would have reached significance at the 5% level). Many CBT reports are published claiming positive outcomes in studies using multiple outcome measures with p values which have not been adjusted, and in which equivocal findings (some measures suggesting rejection of the null hypothesis and others not ) are described as supporting the favoured intervention. It seems to me that those who insist that psychotherapy research can only proceed using the RCT approach are not prepared to accept its judgement. The recent Department of Health review of psychological therapies9 included an allegiance examination of its own work and found CBT practitioners the most likely to favour their own approach and the least likely to consider the possible utility of any other approach. A number of CBT studies appear to provide support for common factors in psychotherapeutic effectiveness10, but this possibility is ignored by CBT researchers8. Perhaps in future studies Sensky and colleagues will have followed the Cochrane Review suggestion of taking more seriously the possible benefits of a “lower grade supportive approach”. CBT’s endorsement of the acute illness/drug research model, which many non-CBT psychotherapists consider inappropriate, gives it a special status in psychotherapy research. Evidence from non RCT investigations, particularly the case reporting approach traditional in psychotherapy, has been dismissed as not scientific and, therefore, irrelevant. CBT is the dominant approach as it reports far more ‘RCTs’ than any other form of psychotherapy, even if the status of many these RCTs is questionable. Perhaps because of allegiance effects, as well as the quantity of CBT studies, CBT also does better than other approaches when others are available for comparison in Cochrane Reviews. For example, CBT is supported by a Cochrane Review for use with Chronic Fatigue Sycndrome11 on the basis of three studies; but when Bulimia Nervosa and Bingeing12 is considered three studies supporting non CBT-psychotherapies are dismissed as insufficient and CBT is once more the favoured therapy. Sensky and Scott1 ask that the challenge is met to understand and respect the critical appraisal of the evidence base. Understanding Cochrane reviews and many of the CBT ‘RCTs’, for example the trial of Sensky and his colleagues8 that I have already mentioned, certainly is a challenge. A criticism of the CBT literature13 is that data are not presented in a clear and straightforward manner. Intricate transformations of the data, partial reporting of the results and complex statistical methods are used which the average clinician is unlikely to be able to follow. If the benefits of CBT are robust and ubiquitous why can this not be demonstrated in a way that is obvious to the reader? The study14 to which Sensky and Scott refer, as if it is definitive evidence, is open to the criticisms above and, if it shows anything clearly, it is not that cognitive therapy is effective in preventing relapse in residual depression rather that an additional psychological intervention improves on medication alone. It might be hoped that the CBT research literature will stimulate those working with different psychotherapeutic approaches to undertake similar studies, including co-operating in multi-arm trials. A major problem is that the CBT literature can also be interpreted as demonstrating the impossibility of conducting RCTs with psychotherapy. I agree that rigorous research methods should be applied to all forms of psychotherapy - but hoping for progress with acceptable evidence confined to that from RCTs is like asking Eriksson to prepare England for the next World Cup only using five-a-side teams and pitches. 1. Sensky T, Scott J. The evidence base of cognitive behavioural therapy. BMJ.com 2002; 7 Feb. 2. Goodman B. Acronym acrimony. Scientific American 2001;285(5):16. 3. Garfield S. Some problems associated with “validated” forms of psychotherapy. Clinical Psychology: Science and Practice 1996; 3: 218- 229. 4. Leff J, Vearnals S, Brewin C, Wolff G, Alexander B, Asen E, et al. The London depression intervention trial. Br J Psychiatry 2001; 177: 95- 100. 5. Index to Abstracts of Cochrane Reviews. The Cochrane Library Issue 3, 2002. www.update-software.com. 6. Cormac I, Jones C, Campbell C. Cognitive behaviour therapy for schizophrenia. The Cochrane Library, Issue 2, 2002. 7. Kinderman P, Cooke A. Recent advances in understanding mental illness and psychotic experiences: a report by the British Psychological Society Division of Clinical Psychology. Leicester: BPS, 2001. 8. Sensky T. Turkington D. Kingdon D. Scott JL. Scott J. Siddle R. O'Carroll M. Barnes TR. A randomized controlled trial of cognitive- behavioral therapy for persistent symptoms in schizophrenia resistant to medication. Archives of General Psychiatry 2000; 57(2):165-72. 9. Department of Health. Treatment Choice in Psychological Therapies and Counselling: Evidence Based Clinical Practice Guideline. London: DOH, 2001. 10. Seligman M. The Effectiveness of Psychotherapy: The Consumer Report Study. American Psychologist 1995; 50: 965-974. 11. Price R, Couper J. Cognitive behaviour therapy for chronic fatigue syndrome in adults. The Cochrane Library, Issue 2, 2002. 12. Hay J, Bacaltchuk J. Psychotherapy for bulimia nervosa and binging. The Cochrane Library, Issue 2, 2002. 13. Johnson D. Peer review of "Cognitive therapy and recovery from acute psychosis". Br J Psychiatry 1996; 169: 608-609. 14. Paykel E, Scott J, Teasdale J, Johnson A, Garland A, Moore R, et al. Prevention of relapse in residual depression by cognitive therapy: a controlled trial. Arch Gen Psychiatry 1999; 56: 829-835.