Intended for healthcare professionals

Papers

# Involving users in the delivery and evaluation of mental health services: systematic review

BMJ 2002; 325 (Published 30 November 2002) Cite this as: BMJ 2002;325:1265
1. Emma L Simpson, research fellow (medelsi{at}south-01.novell.leeds.ac.uk),
2. Allan O House, professor of liaison psychiatry
1. Academic Unit of Psychiatry and Behavioural Sciences, University of Leeds, Leeds LS2 9LT
1. Correspondence to: E L Simpson
• Accepted 14 October 2002

## Abstract

Objectives: To identify evidence from comparative studies on the effects of involving users in the delivery and evaluation of mental health services.

Data sources: English language articles published between January 1966 and October 2001 found by searching electronic databases.

Study selection: Systematic review of randomised controlled trials and other comparative studies of involving users in the delivery or evaluation of mental health services.

Data extraction:Patterns of delivery of services by employees who use or who used to use the service and professional employees and the effects on trainees, research, or clients of mental health services.

Results: Five randomised controlled trials and seven other comparative studies were identified. Half of the studies considered involving users in managing cases. Involving users as employees of mental health services led to clients having greater satisfaction with personal circumstances and less hospitalisation. Providers of services who had been trained by users had more positive attitudes toward users. Clients reported being less satisfied with services when interviewed by users.

Conclusions: Users can be involved as employees, trainers, or researchers without detrimental effect. Involving users with severe mental disorders in the delivery and evaluation of services is feasible.

#### What is already known on this topic

Involving health service users in the NHS is recommended in UK government policy

Involving users in mental health services is generally seen as worthwhile, but the effects of involving users have not been thoroughly evaluated, and few attempts to draw evaluations together have been made

The few comparative studies of users' involvement that have been published indicate that involving users as employees, trainers, or researchers has no negative effect on services and may be of benefit

## Introduction

The Department of Health in the United Kingdom is committed to involving patients in the NHS; it is establishing the Commission for Patient and Public Involvement in Health. Users and carers have been involved in delivering and evaluating mental health services, but the effects of this involvement have not been rigorously assessed.13

We found randomised controlled trials and other comparative studies containing evidence about positive or negative effects of involving users in the delivery or evaluation of mental health services.4 We sought evidence on involving users and the outcomes of involvement on clients (those receiving services). Initially the search encompassed users who were involved in planning services, but we found no comparative studies. We also investigated carers' involvement but found too few studies; only one involved carers as well as users,5 and one other explicitly mentioned a carer's relative with psychiatric history.6

## Methods

We searched Medline, Embase, CINAHL, PsycINFO, HealthSTAR, Cochrane Controlled Trials Register, Web of Science, HMIC, and BIDS for references in English between January 1966 and October 2001 for the terms given in box 1. Searches equivalent to the Medline search were used for other databases.

#### Box 1: Terms used in Medline search

##### MeSH terms

Consumer participation/

Consumer organizations/

Consumer satisfaction/

Caregivers/

Family relations/

Mental disorders/

Mental health/

Mental health services/

Community mental health centers/

Psychiatry/

##### Key and text words ($is a wildcard) user$

consumer$client$

carer$caregiver$

involv$participat$

We wrote to experts and organisations who had an interest in involving healthcare users. We searched the references in all papers for additional studies, whether we included them or not. We searched collections by hand in the Health Sciences Library of the University of Leeds.

### Inclusion and exclusion criteria

We included evaluations of the impact of research on services if users had an active role in the design or in collecting data. We also included studies about users who delivered services by training mental health professionals.

We included studies about delivery involving users in partnership with others if services were integrated by health professionals and users working together in a team; cross-consultation; or recruitment, training, supervision, or payment of users by healthcare providers. We excluded studies which dealt only with the criteria in box 2. Box 3 gives the type of data we extracted.

#### Box 2: Exclusion criteria

We excluded studies if they dealt with only

• Learning disabilities

• Involvement in decisions about a user's own treatment

• Providing information to users

• User satisfaction surveys that were researched by the provider (which do not require users' partnership)

• General health services not specifically aimed at mentally ill people

• Forensic services

• Services for mentally ill people which are not health related, such as housing or vocational rehabilitation

• Services with no contact with professionals or which could not be run by professionals which operate outside the mental health system—for example, self help groups

#### Box 3: Type of data extracted from databases

• Mechanism of involving users, including support available

• Numbers of users involved and diagnoses

• Service or setting of involvement

• Study design, including numbers in comparison groups

• All measures of the process of involving users

• All measures of outcomes for employees who were or who had been users and their clients

To assess the quality of the data, we sought the method of randomisation, evidence of blinding during data collection, and an intention to treat analysis.4 We checked papers for inclusion and exclusion criteria and extracted data onto a standardised form independently by both authors. Meta-analysis was unacceptable because of heterogeneity in the study design and outcome measures so we summarised these qualitatively.4

## Results

We identified five randomised controlled trials and seven other comparative studies.516 Comparisons were mostly of services involving users compared with services with non-users in similar roles. One study compared involvement of more severely disordered users with those less severely disordered14; one study compared lots of contact with involved users with less contact.15

### The nature of users' involvement

Eight studies focused on involving users as service providers, mainly working as case managers in services for clients with severe mental illness (table 1). Case managers need to engage clients, coordinate agencies, and helpmaintain effective delivery; the necessary skills are organisational and interpersonal rather than therapeutic. Two studies looked at the effects of involving users as trainers(table 2), and two studies considered involvingusers as interviewers (table 3).

Table 1

Involving current or former users of mental health services as providers in mental health services

View this table:
Table 2

Involving current or former users of mental health services as trainers of mental health service providers

View this table:
Table 3

Involving current or former users of a mental health service as interviewers of recipients of the service (clients) in evaluating mental health services

View this table:

The users who were involved were current or former users of mental health services who had had serious psychiatric illness—most commonly schizophrenia or bipolar disorder; many had been hospitalised. Employees who were or who had been users of mental health care services and interviewers had similar disorders to their clients.

Interviewers and employees who were or who had been users all received training. Where applicable, this training was similar to that received by employees who had not been users of mental health services. Payment was mentioned in most studies, and support workers were available to nearly all of the employees were or who had been users of services.

### Effects of users' involvement

The process of service delivery of employees who were or who had been users of mental health services differed from that of employees who had not. Users spent longer in supervision,8 in face-to-face contact with clients,17 or doing outreach work,14 and they spent less time on telephone or office work.17 Employees who were or who had been users had a higher turnover rate and had less distinct professional boundaries.8

Employing users in, or alongside, case management services did not have any detrimental effect on clients in terms of symptoms, 7 12 functioning, 5 7 10 12 or quality of life. 5 7 12 Clients of these services had some improved quality of life 10 11; they had fewer reported life problems and improved social functioning. 11 10 Some clients were less of a burden to their families. 5 7 12 In some studies, clients of employees who were or who had been users went for longer until hospital admission and fewer clients needed to be admitted to hospital, 10 11 18 or stay in hospital was shorter,10 although time in hospital was not significantly different in all studies. 5 7 11 13 Services employing people who were or who had been users did not have lower client satisfaction. 5 7 10 12 In one study, clients of employees who were or who had been users were less satisfied with treatment at follow up after one year,19 but they were not after two years.7

Involving users in training gave trainees a more positive attitude toward employees who had been mentally ill and mental illness in general,6 or they looked at users as individuals.15 Clients reported being less satisfied with services when interviewed by other users of the service in evaluation research. 9 16

### Design of study and interpretation

Our review of 298 papers about involving users in delivery of mental health services20 included only 12 comparative studies. We found five randomised trials, only one of which indicated the randomisation method used (alternate allocation according to an alphabetically ordered list of surnames).6 Researchers collecting data were not blinded to treatment group in any of the studies. Four of the trials used intention to treat analysis. 6 7 9 18 Of the other seven studies, researchers were blinded to treatment group in one study.11 No intention to treat analysis was done in these studies.

Some studies were not set up to investigate users' involvement and the results were from a later analysis of routinely collected data.11 Some studies had more than two study groups and did not directly compare involving users with involving those who had not been users.11

Few standardised outcome measures were used unmodified. Measures included adapted versions or selected subscales of existing scales. 5 7 1012 16 Some outcome measures were constructed for the particular study. 6 11 15 Users were involved in the design of a questionnaire developed for one study.9 The use of modified rating scales could have led to bias, as has been shown for unpublished scales.21

Only small numbers of users were involved, with numbers ranging from one user to eight users in a team, making it difficult to apply findings to involving users in general. 6 14 More users were involved in some studies because some users dropped out, generally for unstated reasons, and were replaced. 8 13 17

Sample sizes of studies were small, so estimates of effect were of low power. Clients were not always willing to see staff whom the clients knew had had mental illness.10

Authors interpreted their findings, saying, for example, that when users were less likely to hospitalise clients, it might be because of their own previous bad experiences or because they had more tolerance for behaviour arising from symptoms, used previous experience to help clients stay out of hospital, or more readily engaged with clients needing hospitalisation. 14 18 That interviewers who had been users obtained a higher proportion of negative satisfaction scores might be due to clients feeling more able to be honest with users, thus increasing validity, or it might be that they perceive dissatisfaction as the socially desirable response. 9 16 These possibilities were not explored.

## Discussion

The studies that we identified suggest that users of mental health services can be involved as employees of such services, trainers, or researchers without damaging them. In some studies, benefit was indicated for clients of employees who were or who had been users of services, and, although this was not present across all studies, there were no serious disadvantages. The influence of trainers who had been users on the attitudes of trainees was positive; interviewers who had been users may have brought out negative opinions of services that would not otherwise have been obtained.

Studies suggest that users with a history of severe disorders can be involved in services. This may depend on adequate support, as all of the studies we found included details of the support provided to involved users. This included training and payment for involvement. Service providers have given practical and personal support to users—for example, discussing issues of confidentiality or advising on work matters. 6 17 This support is clearly distinguished from treatment. Our review of non-comparative research supports these findings.20

We found no comparative studies of users' involvement in planning mental health services, but other evaluations of users' involvement in planning in health services—including mental health services—have recently been reviewed.22

Most of the studies we identified involved few users and have substantial methodological weaknesses. Studies of users as service providers mostly originated in the United States and were confined to a case management model. Government policy in the United Kingdom strongly supports the development of involving users in the delivery and evaluation of mental health services. Little evidence exists on the effectiveness of such programmes, and more formal evaluations are needed.

## Acknowledgments

Contributors: AOH was the principal investigator on the project. ELS searched the literature. AOH and ELS decided the inclusion and exclusion criteria, extracted data from included studies, and wrote the report. AOH is guarantor.

## Footnotes

• Funding Non-conditional grant from Leeds Community and Mental Health Services Trust.

• Competing interests None declared.

View Abstract