Evaluation of a palliative care service: problems and pitfallsBMJ 1994; 309 doi: https://doi.org/10.1136/bmj.309.6965.1340 (Published 19 November 1994) Cite this as: BMJ 1994;309:1340
- I R McWhinney,
- M J Bass,
- A Donner
- Thames Valley Family Practice Research Unit, Centre for Studies in Family Medicine, University of Western Ontario, London, Ontario, Canada N6A 5CI.
- Accepted 10 October 1994
Objective: To evaluate a palliative care home support team based on an inpatient unit.
Design: Randomised controlled trial with waiting list. Patients in the study group received the service immediately, those in the control group received it after one month. Main comparison point was at one month. Setting - A city of 300 000 people with a publicly funded home care service and about 200 general practitioners, most of whom provide home care.
Main outcome measures: Pain and nausea levels were measured at entry to trial and at one month, as were quality of life for patients and care givers' health.
Results: Because of early deaths, problems with recruitment, and a low compliance rate for completion of questionnaires, the required sample size was not attained.
Conclusion: In designing evaluations of palliative care services, investigators should be prepared to deal with the following issues: attrition due to early death, opposition to randomisation by patients and referral sources, ethical problems raised by randomisation of dying patients, the appropriate timing of comparison points, and difficulties of collecting data from sick or exhausted patients and care givers. Investigators may choose to evaluate a service from various perspectives using different methods: controlled trials, qualitative studies, surveys, and audits. Randomised trials may prove to be impracticable for evaluation of palliative care.
Because of ethical and methodological difficulties, only three randomised clinical trials of palliative care have been reported
This randomised trial did not succeed in attaining the intended sample size
The chief problems were early deaths, difficulties in recruitment, and a low compliance rate for completion of questionnaires
Other methods for evaluating palliative care services include controlled trials, qualitative studies, surveys, and audits
Many patients drop out of randomised clinical trials, and alternative trial formats have been considered.1 Trials of palliative care (hospice) services present unique difficulties, both ethical and methodological.2 Only three randomised clinical trials of palliative care have been reported*RF 3-5*; two are of home based services.3,5 We report a randomised clinical trial of a palliative care home support team in London, Ontario, that did not succeed in attaining the intended sample size. We report it because an account of the difficulties we encountered may be helpful to those planning similar studies.
The palliative care home support team, based on a 14 bed palliative care unit, consisted of two experienced palliative care nurses (working one week on, one off), one physician, and a part time social worker. Because of the range of home care services available already, the team was planned to be a consulting and support service for family physicians and home care nurses. Within three days of referral by a family doctor or nurse (with family doctor's agreement) one of the team nurses carried out a full assessment in the home. The nurse's assessment and recommendations were discussed with the team doctor, then sent to the family doctor with copies to the visiting nurse and home care case manager. A consultation by the team doctor was available on request. All new and active cases were discussed at the weekly team meeting.
The involvement of the team after the initial assessment depended on the wishes of the patient and family and on negotiation with the family physician and home care nurse. In a few cases there was no further contact with the patient or care givers; in others, progress was followed by telephone calls; in some, a close relationship developed and periodic visits were made to the home. One of the team nurses, with physician back up, was available 24 hours a day, and patients were given a number to call if their home care nurse or family doctor could not be reached. Patients were thus able to get advice from the team or a home visit when problems arose at any time of the day or night.
The evaluation study started after the service had been available for 18 months. It was undertaken to strengthen the case for continuing funding. The randomised design was chosen because of its rigour and also because we felt that it would increase the likelihood of the evaluation being funded. We randomised patients into a “waiting list” group who waited four weeks for assessment by the team and a group given immediate intervention. Emergency consultation by the team physician was made available for patients in the waiting list group if requested by the family physician.
The main outcome measures were pain and nausea, for which the McGill pain questionnaire6 and the Melzack nausea questionnaire7 were used. Other outcome measures were the patient's quality of life,8 and the care giver's health.9,10
The number of patients necessary for the trial was calculated on the basis of a reduction of 33% in the main outcomes of pain and nausea. With an (alpha) level of 0.05 and a ß of 0.20, it was calculated that 110 patients would be required for each group, allowing for 20% attrition. The likely rate of intake and the eligibility criteria were determined by chart review of patients referred to the care team. The eligibility criteria were: aged 18 years or over; being cared for at home by an eligible care giver; having symptomatic cancer which had metastasised or spread to surrounding tissues; and expected to survive for two months. We chose two months as the predicted minimal expectation of life because of the evidence that physicians can make reasonably accurate predictions of survival being greater or less than two months.11
The project coordinator assessed eligibility and conducted randomisation using a computer generated table of random numbers. A research assistant, who was blind to the assignment, visited the home to give more details of the study; obtain written consent; and explain the questionnaires, including the Melzack three day nausea and pain diary, and leave them with the patient and care giver. The assistant visited again after three days to collect the questionnaires then notified the coordinator that baseline data collection was completed. Data collection was repeated at one and two months, one month being the main comparison point.
Within three months of the start of the study, two major problems emerged. A large and unexpected attrition rate (due to death before one month) played a large part in our failure to attain the numbers required for adequate statistical power. Conversely, some patients were not entered in the study because of a predicted early death, then found to be eligible when assessed by the care team. Secondly, the number of eligible referrals varied widely from month to month, with the average being less than predicted. Strenuous efforts were made to attract referrals, including an information sheet for family doctors and presentations to medical meetings. These efforts led to short lived increases, but the average intake for the study as a whole was below the target.
* Randomised clinical trials of palliative care service provide ethical and methodological difficulties - so researchers may wish to use controlled trials, qualitative studies, surveys, or audits for evaluation
Two other problems emerged: admission of the patient to the palliative care unit soon after baseline measurements, either due to inaccurate prognosis or unexpected deterioration; and failure of some patients and care givers to complete the questionnaires at one month. Admission of patients in the control group to the palliative care unit exposed them to a standard of palliative care equivalent to that offered by the palliative care home support team. Failure of patients to complete questionnaires and diary at one month was due to weakness, exhaustion, or cognitive impairment. Failure in care givers was unexplained but is to be expected in people going through a devastating and exhausting experience.
During the trial 307 patients were referred, an average of 3.7 per week. Of these, 141 were ineligible and 20 either refused to participate or died before randomisation. Of the 146 randomised, 53 were lost to follow up before one month, 36 because of early death and 14 because of failure to complete the one month questionnaires. Only 74 care givers completed the questionnaires. There were no clinically or statistically significant differences between the experimental and control groups on any of the measures at one month.
Successful randomised trials are uncommon in health services research. One survey found that only seven out of about 1000 met criteria of adequate design and execution.12 The only trial in the literature comparable to ours had similar problems with patient attrition but did attain its sample size.5 It showed that the services of a palliative care coordinator made little difference to outcome.
Our study encountered insuperable problems with recruitment, death before the comparison point, failure to complete questionnaires, and exposure of control group patients to specialised palliative care. These problems can be attributed to the randomisation process, to factors relating especially to palliative care services, and to health services research in general. Many patients refuse to participate in randomised trials, and referring doctors may be reluctant to participate.1
Our trial posed ethical problems in addition to those inherent in any randomisation process. The main ethical problem raised by randomisation is the denial of a service or treatment to the control group. This is not a problem when there is equipoise between the two interventions being evaluated.13 Since 24 hour coverage by people known to the patient was provided by the home support team but not by any other agency, it was difficult to claim equipoise in our study. Our trial also meant denying some patients a service after it had been available for 18 months. In the evaluation of palliative care services, a waiting list control does not solve the ethical problem. A one month waiting period may not be a burden to patients with a stable condition, but for patients who know that they are dying, even a short wait is too long.
Other problems we encountered are likely to arise in any trial of a palliative care service. As with our patients admitted to the palliative care unit, it may be difficult, under service conditions, to avoid exposure of the control group to the experimental effect. Also, dilution of the experimental effect can occur, as in the study by Kane et al, in which patients randomised to hospice care were cared for in general wards when hospice beds were full.4
Additionally, the outcomes chosen for assessment may not be sensitive to the benefits of the service and the timing of measures may lead to underrecording of the effects of the intervention. In our study, a patient with no pain at baseline and one month might have had a pain crisis between these points which was successfully managed by the team. The support team often spent several hours in patients' homes, dealing with distressing conditions such as restlessness, pain, and dyspnoea in the last 48 hours of patients' lives. Since the team became involved only because the conventional services could not respond, the control group would not have received these services. The timing of measures in our study would not have picked up these effects.
The difficulty patients and care givers have in completing questionnaires is to be expected in palliative care studies. Although it makes for higher costs, trained interviewers can be used to collect data.4 By interpreting questions for patients and care givers, an experienced interviewer can enhance the quality of the responses and take account of factors such as cognitive impairment, exhaustion, and distress. It is difficult, however, for interviewers to remain blind to the patient's group.
The small scale of interventions in palliative care makes it difficult to be certain how much an effect, or lack of effect, is due to the experience, training, personal qualities, and commitment of the team members or to the fit between the people and the role. Because personal services are more difficult to standardise than technical interventions, it is difficult to draw general conclusions, and authors must describe the service in detail.
Randomised trials may prove to be impracticable for palliative care services. When all the problems of randomisation are taken into account, the results of randomised trials are often as difficult to interpret as those of non- randomised controlled studies. The emotional stresses of randomisation on the service providers, and of recruitment on the research team, should not be underestimated. Perhaps this is why so few randomised trials of health services have been reported.
Non-randomised controlled trials using matched controls, with data collected by interview from patients and care givers, have at least two defects: patients selecting hospice care may be different in fundamental ways from those choosing conventional care; and the suitability of controls for hospice care has to be a matter of judgment, rather than part of the normal process of care. Studies using comparisons before and after introduction of a service are also subject to bias from secular change in the health care system. These designs do, however, avoid the problem of the recruitment of patients, and differences between groups can be at least partially controlled by statistical methods, even though the degree to which these techniques can standardise groups is uncertain.14 Before and after studies can be enhanced by a detailed description of the changes occurring during the time of the project.
No single evaluation is likely to be free from flaws. Investigators should consider using different methods in parallel studies. The task of evaluation, and often the chosen method, differs according to the audience being addressed.15 A controlled study could be carried out as well as an audit of records, a survey of users of the service, and a qualitative study of the experiences of providers and recipients. Important decisions are rarely taken on the strength of a single study. To learn as much as possible about a programme, and to provide information for a variety of audiences, it may be desirable to study it from several different perspectives. To help readers to judge it for themselves, investigators should provide as much contextual detail as possible. In planning a study of palliative care, investigators should take into account the likelihood of attrition due to early death, opposition to randomisation by patients and referral sources, ethical problems raised by randomisation of dying patients, the effect of patients' and care givers' exhaustion on collection of data, the possibility of control group patients receiving the intervention, and the importance of the timing of comparison points.
We thank the research assistants who participated in this project, Ms Jennifer McNab and Ms Elizabeth Mayberry, and the research associate, Ms Vanessa Orr. We thank also Ms Reena McDermott and Dr Leslie Levin for their major contribution and Drs Balfour Mount and Mary Vachon for their advice. We are grateful to Ms Lynn Dunikowski and the staff of the Canadian Library of Family Medicine for their bibliographic work and Mrs Bette Cunningham for the preparation of the paper. This study was supported by a grant from the Ontario Ministry of Health, as is the Thames Valley Family Practice Research Unit.