Papers

Involving users in the delivery and evaluation of mental health services: systematic review

Emma L Simpson, research fellow, Allan O House, professor of liaison psychiatry. 

Academic Unit of Psychiatry and Behavioural Sciences, University of Leeds, Leeds LS2 9LT

Correspondence to: E L Simpson
medelsi{at}south-01.novell.leeds.ac.uk

	Abstract

Top Abstract Introduction Methods Results Discussion References

Objectives: To identify evidence from comparative studies on the effects of involving users in the delivery and evaluation of mental health services.
Data sources: English language articles published between January 1966 and October 2001 found by searching electronic databases.
Study selection: Systematic review of randomised controlled trials and other comparative studies of involving users in the delivery or evaluation of mental health services.
Data extraction: Patterns of delivery of services by employees who were current or former users of services and professional employees and the effects on trainees, research, or clients of mental health services.
Results: Five randomised controlled trials and seven other comparative studies were identified. Half of the studies considered involving users in managing cases. Involving users as employees of mental health services led to clients having greater satisfaction with personal circumstances and less hospitalisation. Providers of services who had been trained by users had more positive attitudes toward users. Clients reported being less satisfied with services when interviewed by users.
Conclusions: Users can be involved as employees, trainers, or researchers without detrimental effect. Involving users with severe mental disorders in the delivery and evaluation of services is feasible.

What is already known on this topic Involving health service users in the NHS is recommended in UK government policy Involving users in mental health services is generally seen as worthwhile, but the effects of involving users have not been thoroughly evaluated, and few attempts to draw evaluations together have been made What this study adds The few comparative studies of users' involvement that have been published indicate that involving users as employees, trainers, or researchers has no negative effect on services and may be of benefit

	Introduction

Top Abstract Introduction Methods Results Discussion References

The Department of Health in the United Kingdom is committed to involving patients in the NHS; it is establishing the Commission for Patient and Public Involvement in Health. Users and carers have been involved in delivering and evaluating mental health services, but the effects of this involvement have not been rigorously assessed.^1-3

We sought evidence on involving users and the outcomes of involvement on clients (those receiving services).

	Methods

Top Abstract Introduction Methods Results Discussion References

We searched Medline, Embase, CINAHL, PsycINFO, HealthSTAR, Cochrane Controlled Trials Register, Web of Science, HMIC, and BIDS for references in English between January 1966 and October 2001 (see bmj.com).

We wrote to experts and organisations who had an interest in involving healthcare users asking whether they were aware of additional studies. We searched the references in all papers for additional studies, whether we included them or not. We searched collections by hand in the Health Sciences Library of the University of Leeds.

Exclusion criteria We excluded studies if they dealt with only Learning disabilities Involvement in decisions about a user's own treatment Providing information to users User satisfaction surveys that were researched by the provider (which do not require users' partnership) General health services not specifically aimed at mentally ill people Forensic services Services for mentally ill people which are not health related, such as housing or vocational rehabilitation Services with no contact with professionals or which could not be run by professionals which operate outside the mental health systemfor example, self help groups

Inclusion and exclusion criteria
We included evaluations of the impact of research on services if users had an active role in the design or in collecting data. We also included studies about users who delivered services by training mental health professionals.

We included studies about delivery involving users in partnership with others if services were integrated by health professionals and users working together in a team; cross-consultation; or recruitment, training, supervision, or payment of users by healthcare providers. We excluded studies which dealt only with the criteria in the box.

To assess the quality of the data, we sought the method of randomisation, evidence of blinding during data collection, and an intention to treat analysis.⁴ Meta-analysis was unacceptable because of heterogeneity in the study design and outcome measures so we summarised these qualitatively.⁴

	Results

Top Abstract Introduction Methods Results Discussion References

We identified five randomised controlled trials and seven other comparative studies.^5-16

The nature of users' involvement
Eight studies focused on involving users as service providers, mainly working as case managers in services for clients with severe mental illness (table 1). Case managers need to engage clients, coordinate agencies, and help maintain effective delivery; the necessary skills are organisational and interpersonal rather than therapeutic. Two studies looked at the effects of involving users as trainers (table 2), and two studies considered involving users as interviewers (table 3).

The users who were involved were current or former users of mental health services who had had serious psychiatric illnessmost commonly schizophrenia or bipolar disorder; many had been hospitalised. Employees who were or who had been users of mental health care services and interviewers had similar disorders to their clients.

Interviewers and employees who had been users all received training. Where applicable, this training was similar to that received by employees who had not been users of mental health services. Payment was mentioned in most studies, and support workers were available to nearly all of the employees who were or who had been users of services.

Effects of users' involvement
The process of service delivery of employees who were or who had been users of mental health services differed from that of employees who had not. Users spent longer in supervision, in face-to-face contact with clients, or doing outreach work, and they spent less time on telephone or office work. Employees who were or who had been users had a higher turnover rate and had less distinct professional boundaries.

Employing users in, or alongside, case management services did not have any detrimental effect on clients in terms of symptoms, functioning, or quality of life. Clients of these services had some improved quality of life; they had fewer reported life problems and improved social functioning. Some clients were less of a burden to their families. In some studies, clients of employees who were or who had been users went for longer until hospital admission and fewer clients needed to be admitted to hospital, or stay in hospital was shorter, although time in hospital was not significantly different in all studies. Services employing people who had been users did not have lower client satisfaction. In one study, clients of employees who were or who had been users were less satisfied with treatment at follow up after one year, but they were not after two years.

Involving users in training gave trainees a more positive attitude toward employees who had been mentally ill and mental illness in general, or they looked at users as individuals. Clients reported being less satisfied with services when interviewed by other users of the service in evaluation research.

Design of study and interpretation
Our review of 298 papers about involving users in delivery of mental health services¹⁷ included only 12 comparative studies. We found five randomised trials, only one of which indicated the randomisation method used (alternate allocation according to an alphabetically ordered list of surnames). Researchers collecting data were not blinded to treatment group in any of the studies. Four of the trials used intention to treat analysis. ^{6 7 9 18} Of the other seven studies, researchers were blinded to treatment group in one study.¹¹ No intention to treat analysis was done in these studies.

Some studies were not set up to investigate users' involvement and the results were from a later analysis of routinely collected data.

Few standardised outcome measures were used unmodified. Some outcome measures were constructed for the particular study. Users were involved in the design of a questionnaire developed for one study. The use of modified rating scales could have led to bias, as has been shown for unpublished scales.¹⁹

Only small numbers of users were involved, with numbers ranging from one user to eight users in a team, making it difficult to apply findings to involving users in general. More users were involved in some studies because some users dropped out, generally for unstated reasons, and were replaced.

Sample sizes of studies were small, so estimates of effect were of low power. Clients were not always willing to see staff who the clients knew had had mental illness.

	Discussion

Top Abstract Introduction Methods Results Discussion References

The studies that we identified suggest that users of mental health services can be involved as employees of such services, trainers, or researchers without damaging them. In some studies, benefit was indicated for clients of employees who had been users of services, and, although this was not present across all studies, there were no serious disadvantages. The influence of trainers who had been users on the attitudes of trainees was positive; interviewers who had been users may have brought out negative opinions of services that would not otherwise have been obtained.

We found no comparative studies of users' involvement in planning mental health services, but other evaluations of users' involvement in planning in health servicesincluding mental health serviceshave recently been reviewed.²⁰

Most of the studies we identified involved few users and have substantial methodological weaknesses. Studies of users as service providers mostly originated in the United States and were confined to a case management model. Government policy in the United Kingdom strongly supports the development of involving users in the delivery and evaluation of mental health services. Little evidence exists on the effectiveness of such programmes, and more formal evaluations are needed.

	Acknowledgments

Contributors: See bmj.com

	Footnotes

Funding: Non-conditional grant from Leeds Community and Mental Health Services Trust.

Competing interests: None declared.

This is an abridged version; the full version is on bmj.com

	References

Top Abstract Introduction Methods Results Discussion References

1.	Mental Health Task Force User Group. Forging our futures: lighting the fire. London: Department of Health, 1995.
2.	NHS Health Advisory Service. Voices in partnership: involving users and carers in commissioning and delivering mental health services. London: Stationery Office, 1997.
3.	Department of Health. National service framework for mental health modern standards and service models. London: DoH, 1999.
4.	NHS Centre for Reviews and Dissemination. Undertaking systematic reviews of research on effectiveness. CRD , 2001.
5.	O'Donnell M, Parker G, Proberts M, Matthews R, Fisher D, Johnson B, et al. A study of client-focused case management and consumer advocacy: the community and consumer service project. Aust N Z J Psychiatry 1999; 33: 684-693[CrossRef][Web of Science][Medline].
6.	Cook JA, Jonikas JA, Razzano L. A randomized evaluation of consumer versus nonconsumer training of state mental health service providers. Community Ment Health J 1995; 31: 229-238[CrossRef][Web of Science][Medline].
7.	Solomon P, Draine J. The efficacy of a consumer case management team: 2-year outcomes of a randomized trial. J Ment Health Adm 1995; 22: 135-146[Web of Science][Medline].
8.	Paulson R, Herinckx H, Demmler J, Clarke G, Cutter D, Birecree E. Comparing practice patterns of consumer and non-consumer mental health service providers. Community Ment Health J 1999; 35: 251-269[CrossRef][Web of Science][Medline].
9.	Clark CC, Scott EA, Boydell KM, Goering P. Effects of client interviewers on client-reported satisfaction with mental health services. Psychiatr Serv 1999; 50: 961-963[Abstract/Free Full Text].
10.	Klein AR, Cnaan RA, Whitecraft J. Significance of peer support with dually diagnosed clients: findings from a pilot study. Res Soc Work Pract 1998; 8: 529-551[Abstract/Free Full Text].
11.	Felton CJ, Stastny P, Shern DL, Blanch A, Donahue SA, Knight E, et al. Consumers as peer specialists on intensive case management teams: impact on client outcomes. Psychiatr Serv 1995; 46: 1037-1044[Abstract/Free Full Text].
12.	Chinman MJ, Rosenheck R, Lam JA, Davidson L. Comparing consumer and nonconsumer provided case management services for homeless persons with serious mental illness. J Nerv Ment Dis 2000; 188: 446-453[CrossRef][Web of Science][Medline].
13.	Chinman MJ, Weingarten R, Stayner D, Davidson L. Chronicity reconsidered: improving person-environment fit through a consumer-run service. Community Ment Health J 2001; 37: 215-229[CrossRef][Web of Science][Medline].
14.	Lyons JS, Cook JA, Ruth AR, Karver M, Slagg NB. Service delivery using consumer staff in a mobile crisis assessment program. Community Ment Health J 1996; 32: 33-40[CrossRef][Web of Science][Medline].
15.	Wood J, Wilson-Barnett J. The influence of user involvement on the learning of mental health nursing students. NT Research 1999; 4: 257-270.
16.	Polowczyk D, Brutus M, Orvieto AA, Vidal J, Cipriani D. Comparison of patient and staff surveys of consumer satisfaction. Hosp Community Psychiatry 1993; 44: 589-591[Abstract/Free Full Text].
17.	Simpson EL, House AO, Barkham M. A guide to involving users, ex-users and carers in mental health service planning, delivery or research: a health technology approach. Leeds: Academic Unit of Psychiatry and Behavioural Sciences, University of Leeds, 2002.
18.	Clarke GN, Herinckx HA, Kinney RF, Paulson RI, Cutler DL, Lewis K, et al. Psychiatric hospitalizations, arrests, emergency room visits, and homelessness of clients with serious and persistent mental illness: findings from a randomized trial of two ACT programs vs. usual care. Ment Health Serv Res 2000; 2: 155-164[CrossRef][Medline].
19.	Marshall M, Lockwood A, Bradley C, Adams C, Joy C, Fenton M. Unpublished rating scales: a major source of bias in randomised controlled trials of treatments for schizophrenia. Br J Psychiatry 2000; 176: 249-252[Abstract/Free Full Text].
20.	Crawford MJ, Rutter D, Manley C, Weaver T, Bhui K, Fulop N, et al. Systematic review of involving patients in the planning and development of health care. BMJ 2002; 325: 1263-1265[Abstract/Free Full Text].
21.	O'Donnell M, Proberts M, Parker G. Development of a consumer advocacy program. Aust N Z J Psych 1998; 32: 873-879[Web of Science][Medline].

(Accepted 14 October 2002)