Randomised comparison of the effectiveness and costs of community and hospital based mental health services for children with behavioural disordersBMJ 2000; 321 doi: https://doi.org/10.1136/bmj.321.7268.1047 (Published 28 October 2000) Cite this as: BMJ 2000;321:1047
- Richard Harrington, professor ()a,
- Sarah Peters, research associatea,
- Jonathan Green, senior lecturera,
- Sarah Byford, research fellowb,
- Jane Woods, research assistanta,
- Ruth McGowan, specialist registrara
- a Department of Child and Adolescent Psychiatry, University of Manchester, Manchester M27 4HA,
- b Centre for Health Economics, University of York, York YO1 5DD
- Correspondence to: R Harrington, University Department of Child and Adolescent Psychiatry, Royal Manchester Children's Hospital, Pendlebury, Manchester M27 4HA
- Accepted 1 August 2000
Objective: To test the hypothesis that a community based intervention by secondary child and adolescent mental health services would be significantly more effective and less costly than a hospital based intervention.
Design: Open study with two randomised parallel groups.
Setting: Two health districts in the north of England.
Participants: Parents of 3 to 10 year old children with behavioural disorder who had been referred to child and adolescent mental health services.
Intervention: Parental education groups.
Main outcome measures: Parents' and teachers' reports of the child's behaviour, parental depression, parental criticism of the child, impact of the child's behaviour on the family.
Results: 141 subjects were randomised to community (n=72) or hospital (n=69) treatment. Primary outcome data were obtained on 115 (82%) cases a year later. Intention to treat analyses showed no significant differences between the community and hospital based groups on any of the outcome measures, or on costs. Parental depression was common and predicted the child's outcome.
Conclusions: Location of child mental health services may be less important than the range of services that they provide, which should include effective treatment for parents' mental health problems.
Views about where to provide secondary mental health services for children and adolescents have changed repeatedly over the past 50 years. Child psychiatry started as a community discipline in child guidance clinics.1 There were, however, many practical difficulties in the administration of these clinics,1 and during the 1970s and '80s many clinics closed and were replaced by hospital based services. 1 2 Over the past 15 years, however, hospital based services too have been criticised—on the grounds that they are inaccessible, stigmatising, expensive, poorly integrated with community services, and less likely to produce gains that generalise to other environments, such as school. 3 4 Political pressure has also been applied to child mental health services to return to the community.2
The assumption that community based child mental health services lead to better outcomes than hospital based services has not been tested in a randomised trial in the United Kingdom. We conducted such a trial, whose main hypothesis was that for children with behavioural disorders a community based intervention would be significantly more effective and less costly than a hospital based intervention.
Participants and methods
The study was based on the parents of children with behavioural disorders. Parents were eligible if they were judged able to participate in groups—for example, if they did not have a major mental disorder—and had children who were (a) aged 3-10 years, (b) had a clinical diagnosis of oppositional disorder,5 (c) had normal intelligence (clinical judgment), and (d) had been referred to the child and adolescent mental health service in either of two health districts in the north of England.
During the study each district child and adolescent mental health service provided the same intervention either in a children's hospital or in a community setting—for example, health centres and community resource centres. As this was a pragmatic trial of the interventions used in the NHS, each service used their routine interventions for behavioural disorder for the age group being studied. In one of the districts, this was the videotape modelling parental group education programme of Webster-Stratton and colleagues.6 The other district used a programme of parental education groups with parallel child groups. In both districts the interventions were provided by various professionals, including community psychiatric nurses, psychologists, social workers, and psychiatrists. Therapists who took part in the study were trained in at least two parental groups before leading a group.
Procedures for assignment and blinding
The unit of randomisation was the parent/index child pair. After written consent had been obtained, an independent statistician at a distant site randomly allocated participants to the community or hospital based interventions, stratified by health district. Group allocation was concealed from the outcome assessor, who was asked at the end of the study to guess which intervention had been given to which parent. These guesses were no better than chance (59/141 or 42% correct). As studies of psychosocial treatments can only ever be single blind, participants' expectancies of treatment could bias the results. The parents' expectancy of treatment after randomisation was therefore assessed with a 0 to 8 continuous scale, where 8 was a very high expectancy that treatment would help.7 The mean level of expectancy among parents was similar in both groups (community group: 5.7 (95% confidence interval 5.3 to 6.1); hospital group: 5.8 (5.4 to 6.3)).
Assessment of effectiveness
Measures were completed before treatment and at two follow up stages—three months after treatment started and at about one year. Because parental perception of a child's problems is one of the most important determinants of the use of services,8 the primary outcome was parental report of the child's behaviour.9 Secondary outcomes comprised the teacher's report of the child's behaviour,9 parental reports of the impact of the child's behaviour on the family,10 parental criticism of the child (assessed by counting the number of critical comments during a five minute speech sample11), and parental perception of parenting problems.12 Parental depression was assessed with the Beck questionnaire.13 In line with previous research in the United Kingdom with the parents of children with behavioural disorder,14 a score of 15 or more on the Beck scale was categorised as “high,” indicating a high level of maternal depression.
Assessment of costs
Information on the use of all services by both the children and their primary carer (usually the mother) during the trial was collected from the primary carer at the final follow up interview. A questionnaire was designed for the purpose of the study and based on data collection methods developed in a previous trial.15 The perspective of the trial was that of all service providers, including the NHS, social services departments, education departments, and voluntary and private sectors. This enabled the differential impact of the parenting skills groups on each sector to be clearly quantified. In addition, the cost of travel to attend the sessions and the cost of the crèche facilities were recorded. Unit costs were for the financial year 1998-9 and were collected from local service providers or national published unit costs16–18 or calculated directly from relevant salary scales. All future costs were discounted at an annual rate of 5%. Further details on the costing of services are available on request.
We projected a sample size of about 40 cases in each group for two reasons. Firstly, before starting the study we had asked the purchaser (the health authority) and the provider of the existing child and adolescent mental health service in one of the districts (Salford) how large the difference in treatment effect between a community and hospital service would have to be to influence their plans for child and adolescent mental health services. These services were being reviewed at the time of the study, and both the purchaser and the provider made a commitment to implement the findings. They agreed that only a large difference (effect size of 0.8 for the mean difference in parental report of child behaviour between community and hospital treatment at the 3 month assessment) would lead to changes in the ways that services were delivered. Thirty four cases per group are needed for a 90% chance of detecting this difference with a two sided test at the conventional 5% significance level.19 Secondly, a Canadian study comparing community and clinic based child mental health services found a significant difference in parental reports of child behaviour with about 40 cases in each group.4
Just one intention to treat analysis—that is, on all participants who were randomised and who could be followed up regardless of whether they had started or completed the intervention—was conducted at the end of the study. The data were analysed with SPSS 9.0 for Windows. Changes from baseline were calculated for the outcomes, and t tests for independent samples were used to compare the community and hospital groups.
Although costs were not normally distributed, analyses compared the mean costs in the two groups with standard t test methods, with the validity of results confirmed by using the “bootstrapping” technique.20 The advantage of this approach, rather than logarithmic transformation or conventional non-parametric tests, is the ability to make inferences about the arithmetic mean.21
Participant flow and follow up
The figure shows the trial profile. In all, 187 eligible parents were referred to the trial. Of these, 25 failed to respond to an invitation to take part—for example, did not attend—and 21 refused randomisation. The remaining 141 parents were then randomly allocated to community (n=72) or hospital (n=69) treatment; 115/141 (82%) cases had complete data at baseline and at follow up one year later.
Participants' and children's characteristics
The median age of the children was 6.9 (range 3-10) years, and 112 (79%) were boys. The primary carer, who completed the parental questionnaires, was the mother in 136 out of 141 cases. In 94 out of 141 (67%) cases the parents were receiving state benefits, and in 61 out of 141 (43%) cases the parents were single. Seventy three out of 134 (54%) parents who completed the Beck depression questionnaire at baseline had a “high” score. The groups did not differ significantly in respect of these characteristics.
The mean behaviour intensity score (possible range 36-252, where the higher the score, the greater the intensity of behavioural problems)9 before treatment was 178.0 (95% confidence interval 172.8 to 183.1), higher than the mean score before treatment reported in another UK study (152.8; 144.3 to 161.4)22 and in studies of parental education in North America.23
The two groups did not differ significantly on any of the measures at baseline or at either of the two follow up assessments (table 1). There were no significant effects of location of treatment (community v hospital) on changes in any of the outcomes (table 2). A high proportion of children (89/116 (77%)) had a high score (≥127) on parental report of the intensity of behavioural problems at the three month assessment. The risk of children having a high score after treatment was greater (odds ratio 2.8; 1.3 to 6.1) for those whose primary carer had a high score on the Beck depression questionnaire at baseline (48/55) compared with those with a low score (≤14) (36/56). Confounding by social class or behavioural problems at baseline did not account for this association.
In line with other research with the families of children with behavioural disorders, 22 24 there were high rates of non-attendance. Location did not influence attendance rate. For example, the proportion of participants who attended more than 50% of offered sessions was similar in the community (37/72 (51%)) and hospital (41/69 (59%)) groups (odds ratio 0.8; 0.6 to 1.2). Compliance with treatment tended to be worse (0.7; 0.5 to 1.1) among parents who were depressed at baseline (37/73 (51%)) than among those who were not (40/61 (66%)). The relation between parental depression at baseline and children's behavioural problems at the three month assessment was stronger among those who complied poorly with treatment (5.8; 1.5 to 23.2) than among those who complied well (1.8; 0.7 to 4.7).
This study followed up 82% of randomised participants more than a year later. Nevertheless, missing outcomes can violate the principles of intention to treat analysis.25 We therefore conducted an extreme case analysis in which missing outcomes on community cases were assumed to have improved (to the mean value for the community group at three months), and in which missing outcomes in the hospital group were assumed not to have changed. This did not significantly affect the results.
Altogether, 118 out of 141 (84%) parents (61 in the community group, 57 in the hospital group) completed the resource questionnaire at the final follow up interview; these responses were included in the economic evaluation (table 3). The mean length of follow up was similar in both groups (65.7 weeks in the community group, 65.0 in the hospital group). To remove the influence of this small difference, however, costs per week (as well as overall cost) were calculated. No significant differences between the hospital and community groups were found in the mean overall cost per child (difference £904 (£−1254 to £3062); P=0.41), per primary carer (£611 (£−143 to £1365); P=0.11), or in total (£1515 (£−742 to £3772); P=0.19), or in terms of cost per week. To assess the robustness and generalisability of the results, several univariate sensitivity analyses were carried out. These analyses did not affect the results.
The present study did not find that community based treatment was more effective than hospital based treatment. This finding contrasts with that of Cunningham and colleagues4 in Canada, who reported that children with behavioural problems treated in the community had better outcomes than those referred to a specialised clinic. It is, however, difficult to interpret the results of that study because outcome data were obtained on less than a third of randomised subjects.
Three issues should be borne in mind when interpreting the results of the present study. Firstly, the trial was powered on the basis of a significant difference in clinical outcomes. The sample size may have been too small to detect a significant difference in costs. Indeed, the actual cost differences found between the two groups were large, with the hospital group costing 30% less overall than the community group. Secondly, there are several different models of community child and adolescent mental health services. This study modelled the common situation in which secondary services were located in just one or two community settings. Different results might have been obtained if we had studied primary care interventions, such as training health visitors in parental education methods,26 or interventions in which mental health professionals support community schemes such as befriending.27 These interventions have, however, seldom been evaluated in randomised trials in the United Kingdom. Thirdly, the results from this study may not apply to the treatment of other child psychiatric disorders or, indeed, to the treatment of behavioural problems using other methods.
Planning services for children with behavioural disorders
We conclude that in planning services for children with behavioural disorders, greater attention must be paid to factors other than the location. Our findings suggest, for example, that a service for children with behavioural problems must also be able to call not just on parental education groups but also on interventions such as effective treatments for parental depression. The finding that parental depression was more strongly associated with poor child outcomes in the presence of poor compliance with treatment suggests, however, that poor compliance might be partly responsible for the failure of treatment in the children of depressed parents. This group may therefore need extra help to attend child mental health services. As a substantial proportion of the costs associated with behavioural disorders in children are borne by the educational services, there may also be scope for further developing the role of mental health workers in schools.
What is already known on this topic
It is assumed that community based child mental health services lead to better outcomes than hospital based services, although this has not been tested in a randomised trial in the United Kingdom
What this study adds
Community based child mental health services are not necessarily more effective or cheaper than hospital based services
The outcomes of children's mental health problems are determined by many other factors, such as parental mental health
Child mental health services should provide effective treatment for parental mental health problems
The range of mental health services available is more important than where the service is given
We thank Fiona Campbell and Daphne Kounali of the department of statistics at Hope Hospital, Salford, who conducted the randomisation, and the therapists who conducted the parental education groups; Chrissie Pickin of Salford and Trafford Health Authority; and Sibyl Zaden of the University of California, who rated the five minute speech samples.
Contributors: RH and JG had the idea for the study, obtained the grant, and managed the project. RH conducted the analyses and wrote up the study. SP and JW conducted the outcome assessments, contributed to the design, and revised the final manuscript. SB designed the economic evaluation and conducted the economic analysis. RMcG conducted many of the parental groups and revised the manuscript. RH will act as guarantor for the paper.
Funding Research grant from the NHS Executive's Motherhood and Child Health initiative. Clinical costs were provided by Manchester Children's Hospital NHS Trust (funding for two of the therapists to train in the Webster-Stratton programme in Seattle, United States) and by Salford social services.
Competing interests None declared.