Papers

Involving users in the delivery and evaluation of mental health services: systematic review

Emma L Simpson, research fellow, Allan O House, professor of liaison psychiatry. 

Academic Unit of Psychiatry and Behavioural Sciences, University of Leeds, Leeds LS2 9LT

Correspondence to: E L Simpson
medelsi{at}south-01.novell.leeds.ac.uk

	Abstract

Top Abstract Introduction Methods Results Discussion References

Objectives: To identify evidence from comparative studies on the effects of involving users in the delivery and evaluation of mental health services.
Data sources: English language articles published between January 1966 and October 2001 found by searching electronic databases.
Study selection: Systematic review of randomised controlled trials and other comparative studies of involving users in the delivery or evaluation of mental health services.
Data extraction: Patterns of delivery of services by employees who use or who used to use the service and professional employees and the effects on trainees, research, or clients of mental health services.
Results: Five randomised controlled trials and seven other comparative studies were identified. Half of the studies considered involving users in managing cases. Involving users as employees of mental health services led to clients having greater satisfaction with personal circumstances and less hospitalisation. Providers of services who had been trained by users had more positive attitudes toward users. Clients reported being less satisfied with services when interviewed by users.
Conclusions: Users can be involved as employees, trainers, or researchers without detrimental effect. Involving users with severe mental disorders in the delivery and evaluation of services is feasible.

What is already known on this topic Involving health service users in the NHS is recommended in UK government policy Involving users in mental health services is generally seen as worthwhile, but the effects of involving users have not been thoroughly evaluated, and few attempts to draw evaluations together have been made What this study adds The few comparative studies of users' involvement that have been published indicate that involving users as employees, trainers, or researchers has no negative effect on services and may be of benefit

	Introduction

Top Abstract Introduction Methods Results Discussion References

The Department of Health in the United Kingdom is committed to involving patients in the NHS; it is establishing the Commission for Patient and Public Involvement in Health. Users and carers have been involved in delivering and evaluating mental health services, but the effects of this involvement have not been rigorously assessed.^1-3

We found randomised controlled trials and other comparative studies containing evidence about positive or negative effects of involving users in the delivery or evaluation of mental health services.⁴ We sought evidence on involving users and the outcomes of involvement on clients (those receiving services). Initially the search encompassed users who were involved in planning services, but we found no comparative studies. We also investigated carers' involvement but found too few studies; only one involved carers as well as users,⁵ and one other explicitly mentioned a carer's relative with psychiatric history.⁶

	Methods

Top Abstract Introduction Methods Results Discussion References

We searched Medline, Embase, CINAHL, PsycINFO, HealthSTAR, Cochrane Controlled Trials Register, Web of Science, HMIC, and BIDS for references in English between January 1966 and October 2001 for the terms given in box 1. Searches equivalent to the Medline search were used for other databases.

We wrote to experts and organisations who had an interest in involving healthcare users. We searched the references in all papers for additional studies, whether we included them or not. We searched collections by hand in the Health Sciences Library of the University of Leeds.

Inclusion and exclusion criteria
We included evaluations of the impact of research on services if users had an active role in the design or in collecting data. We also included studies about users who delivered services by training mental health professionals.

We included studies about delivery involving users in partnership with others if services were integrated by health professionals and users working together in a team; cross-consultation; or recruitment, training, supervision, or payment of users by healthcare providers. We excluded studies which dealt only with the criteria in box 2. Box 3 gives the type of data we extracted.

To assess the quality of the data, we sought the method of randomisation, evidence of blinding during data collection, and an intention to treat analysis.⁴ We checked papers for inclusion and exclusion criteria and extracted data onto a standardised form independently by both authors. Meta-analysis was unacceptable because of heterogeneity in the study design and outcome measures so we summarised these qualitatively.⁴

	Results

Top Abstract Introduction Methods Results Discussion References

We identified five randomised controlled trials and seven other comparative studies.^5-16 Comparisons were mostly of services involving users compared with services with non-users in similar roles. One study compared involvement of more severely disordered users with those less severely disordered¹⁴; one study compared lots of contact with involved users with less contact.¹⁵

The nature of users' involvement
Eight studies focused on involving users as service providers, mainly working as case managers in services for clients with severe mental illness (table 1). Case managers need to engage clients, coordinate agencies, and help maintain effective delivery; the necessary skills are organisational and interpersonal rather than therapeutic. Two studies looked at the effects of involving users as trainers (table 2), and two studies considered involving users as interviewers (table 3).

The users who were involved were current or former users of mental health services who had had serious psychiatric illnessmost commonly schizophrenia or bipolar disorder; many had been hospitalised. Employees who were or who had been users of mental health care services and interviewers had similar disorders to their clients.

Interviewers and employees who were or who had been users all received training. Where applicable, this training was similar to that received by employees who had not been users of mental health services. Payment was mentioned in most studies, and support workers were available to nearly all of the employees were or who had been users of services.

Effects of users' involvement
The process of service delivery of employees who were or who had been users of mental health services differed from that of employees who had not. Users spent longer in supervision,⁸ in face-to-face contact with clients,¹⁷ or doing outreach work,¹⁴ and they spent less time on telephone or office work.¹⁷ Employees who were or who had been users had a higher turnover rate and had less distinct professional boundaries.⁸

Employing users in, or alongside, case management services did not have any detrimental effect on clients in terms of symptoms, ^{7 12} functioning, ^{5 7 10 12} or quality of life. ^{5 7 12} Clients of these services had some improved quality of life ^{10 11} ; they had fewer reported life problems and improved social functioning. ^{11 10} Some clients were less of a burden to their families. ^{5 7 12} In some studies, clients of employees who were or who had been users went for longer until hospital admission and fewer clients needed to be admitted to hospital, ^{10 11 18} or stay in hospital was shorter,¹⁰ although time in hospital was not significantly different in all studies. ^{5 7 11 13} Services employing people who were or who had been users did not have lower client satisfaction. ^{5 7 10 12} In one study, clients of employees who were or who had been users were less satisfied with treatment at follow up after one year,¹⁹ but they were not after two years.⁷

Involving users in training gave trainees a more positive attitude toward employees who had been mentally ill and mental illness in general,⁶ or they looked at users as individuals.¹⁵ Clients reported being less satisfied with services when interviewed by other users of the service in evaluation research. ^{9 16}

Design of study and interpretation
Our review of 298 papers about involving users in delivery of mental health services²⁰ included only 12 comparative studies. We found five randomised trials, only one of which indicated the randomisation method used (alternate allocation according to an alphabetically ordered list of surnames).⁶ Researchers collecting data were not blinded to treatment group in any of the studies. Four of the trials used intention to treat analysis. ^{6 7 9 18} Of the other seven studies, researchers were blinded to treatment group in one study.¹¹ No intention to treat analysis was done in these studies.

Some studies were not set up to investigate users' involvement and the results were from a later analysis of routinely collected data.¹¹ Some studies had more than two study groups and did not directly compare involving users with involving those who had not been users.¹¹

Few standardised outcome measures were used unmodified. Measures included adapted versions or selected subscales of existing scales. ^{5 7 10-12 16} Some outcome measures were constructed for the particular study. ^{6 11 15} Users were involved in the design of a questionnaire developed for one study.⁹ The use of modified rating scales could have led to bias, as has been shown for unpublished scales.²¹

Only small numbers of users were involved, with numbers ranging from one user to eight users in a team, making it difficult to apply findings to involving users in general. ^{6 14} More users were involved in some studies because some users dropped out, generally for unstated reasons, and were replaced. ^{8 13 17}

Sample sizes of studies were small, so estimates of effect were of low power. Clients were not always willing to see staff whom the clients knew had had mental illness.¹⁰

Authors interpreted their findings, saying, for example, that when users were less likely to hospitalise clients, it might be because of their own previous bad experiences or because they had more tolerance for behaviour arising from symptoms, used previous experience to help clients stay out of hospital, or more readily engaged with clients needing hospitalisation. ^{14 18} That interviewers who had been users obtained a higher proportion of negative satisfaction scores might be due to clients feeling more able to be honest with users, thus increasing validity, or it might be that they perceive dissatisfaction as the socially desirable response. ^{9 16} These possibilities were not explored.

	Discussion

Top Abstract Introduction Methods Results Discussion References

The studies that we identified suggest that users of mental health services can be involved as employees of such services, trainers, or researchers without damaging them. In some studies, benefit was indicated for clients of employees who were or who had been users of services, and, although this was not present across all studies, there were no serious disadvantages. The influence of trainers who had been users on the attitudes of trainees was positive; interviewers who had been users may have brought out negative opinions of services that would not otherwise have been obtained.

Studies suggest that users with a history of severe disorders can be involved in services. This may depend on adequate support, as all of the studies we found included details of the support provided to involved users. This included training and payment for involvement. Service providers have given practical and personal support to usersfor example, discussing issues of confidentiality or advising on work matters. ^{6 17} This support is clearly distinguished from treatment. Our review of non-comparative research supports these findings.²⁰

We found no comparative studies of users' involvement in planning mental health services, but other evaluations of users' involvement in planning in health servicesincluding mental health serviceshave recently been reviewed.²²

Most of the studies we identified involved few users and have substantial methodological weaknesses. Studies of users as service providers mostly originated in the United States and were confined to a case management model. Government policy in the United Kingdom strongly supports the development of involving users in the delivery and evaluation of mental health services. Little evidence exists on the effectiveness of such programmes, and more formal evaluations are needed.

	Acknowledgments

Contributors: AOH was the principal investigator on the project. ELS searched the literature. AOH and ELS decided the inclusion and exclusion criteria, extracted data from included studies, and wrote the report. AOH is guarantor.

	Footnotes

Funding: Non-conditional grant from Leeds Community and Mental Health Services Trust.

Competing interests: None declared.

	References

Top Abstract Introduction Methods Results Discussion References

1.	Mental Health Task Force User Group. Forging our futures: lighting the fire. London: Department of Health, 1995.
2.	NHS Health Advisory Service. Voices in partnership: involving users and carers in commissioning and delivering mental health services. London: Stationery Office, 1997.
3.	Department of Health. National service framework for mental health modern standards and service models. London: DoH, 1999.
4.	NHS Centre for Reviews and Dissemination. Undertaking systematic reviews of research on effectiveness. CRD , 2001.
5.	O'Donnell M, Parker G, Proberts M, Matthews R, Fisher D, Johnson B, et al. A study of client-focused case management and consumer advocacy: the community and consumer service project. Aust N Z J Psychiatry 1999; 33: 684-693[CrossRef][ISI][Medline].
6.	Cook JA, Jonikas JA, Razzano L. A randomized evaluation of consumer versus nonconsumer training of state mental health service providers. Community Ment Health J 1995; 31: 229-238[CrossRef][ISI][Medline].
7.	Solomon P, Draine J. The efficacy of a consumer case management team: 2-year outcomes of a randomized trial. J Ment Health Adm 1995; 22: 135-146[ISI][Medline].
8.	Paulson R, Herinckx H, Demmler J, Clarke G, Cutter D, Birecree E. Comparing practice patterns of consumer and non-consumer mental health service providers. Community Ment Health J 1999; 35: 251-269[CrossRef][ISI][Medline].
9.	Clark CC, Scott EA, Boydell KM, Goering P. Effects of client interviewers on client-reported satisfaction with mental health services. Psychiatr Serv 1999; 50: 961-963[Abstract/Free Full Text].
10.	Klein AR, Cnaan RA, Whitecraft J. Significance of peer support with dually diagnosed clients: findings from a pilot study. Res Soc Work Pract 1998; 8: 529-551[Abstract/Free Full Text].
11.	Felton CJ, Stastny P, Shern DL, Blanch A, Donahue SA, Knight E, et al. Consumers as peer specialists on intensive case management teams: impact on client outcomes. Psychiatr Serv 1995; 46: 1037-1044[Abstract/Free Full Text].
12.	Chinman MJ, Rosenheck R, Lam JA, Davidson L. Comparing consumer and nonconsumer provided case management services for homeless persons with serious mental illness. J Nerv Ment Dis 2000; 188: 446-453[CrossRef][ISI][Medline].
13.	Chinman MJ, Weingarten R, Stayner D, Davidson L. Chronicity reconsidered: improving person-environment fit through a consumer-run service. Community Ment Health J 2001; 37: 215-229[CrossRef][ISI][Medline].
14.	Lyons JS, Cook JA, Ruth AR, Karver M, Slagg NB. Service delivery using consumer staff in a mobile crisis assessment program. Community Ment Health J 1996; 32: 33-40[CrossRef][ISI][Medline].
15.	Wood J, Wilson-Barnett J. The influence of user involvement on the learning of mental health nursing students. NT Research 1999; 4: 257-270.
16.	Polowczyk D, Brutus M, Orvieto AA, Vidal J, Cipriani D. Comparison of patient and staff surveys of consumer satisfaction. Hosp Community Psychiatry 1993; 44: 589-591[Abstract/Free Full Text].
17.	Solomon P, Draine J. Service delivery differences between consumer and non-consumer case managers in mental health. Res Social Work Pract 1996; 6: 193-207.
18.	Clarke GN, Herinckx HA, Kinney RF, Paulson RI, Cutler DL, Lewis K, et al. Psychiatric hospitalizations, arrests, emergency room visits, and homelessness of clients with serious and persistent mental illness: findings from a randomized trial of two ACT programs vs. usual care. Ment Health Serv Res 2000; 2: 155-164[CrossRef][Medline].
19.	Solomon P, Draine J. One year outcome of a randomized trial of consumer case managers. Eval and Program Plann 1995; 18: 117-127.
20.	Simpson EL, House AO, Barkham M. A guide to involving users, ex-users and carers in mental health service planning, delivery or research: a health technology approach. Leeds: Academic Unit of Psychiatry and Behavioural Sciences, University of Leeds, 2002.
21.	Marshall M, Lockwood A, Bradley C, Adams C, Joy C, Fenton M. Unpublished rating scales: a major source of bias in randomised controlled trials of treatments for schizophrenia. Br J Psychiatry 2000; 176: 249-252[Abstract/Free Full Text].
22.	Crawford MJ, Rutter D, Manley C, Weaver T, Bhui K, Fulop N, et al. Systematic review of involving patients in the planning and development of health care. BMJ 2002; 325: 1263-1265[Abstract/Free Full Text].
23.	O'Donnell M, Proberts M, Parker G. Development of a consumer advocacy program. Aust N Z J Psych 1998; 32: 873-879[ISI][Medline].

(Accepted 14 October 2002)