An investigation into general practitioners associated with high patient mortality flagged up through the Shipman inquiry: retrospective analysis of routine dataBMJ 2004; 328 doi: https://doi.org/10.1136/bmj.328.7454.1474 (Published 17 June 2004) Cite this as: BMJ 2004;328:1474
- Mohammed A Mohammed, senior research fellow ()1,
- Anthony Rathbone, primary care medical adviser2,
- Paulette Myers, head of clinical governance3,
- Divya Patel, research analyst4,
- Helen Onions, health intelligence manager2,
- Andrew Stevens, professor1
- 1Department of Public Health and Epidemiology, University of Birmingham, Edgbaston, Birmingham B15 2TT
- 2Telford and Wrekin Primary Care Trust, Telford TF1 5RY
- 3Shropshire and Staffordshire Strategic Health Authority, Stafford ST16 3SR
- 4Shropshire County Primary Care Trust, Shrewsbury SY3 8XL
- Correspondence to: M A Mohammed
- Accepted 27 April 2004
Objective To identify a credible explanation for the excessively high mortality associated with general practitioners who were flagged up by the Shipman inquiry.
Design Retrospective analysis of routine data.
Setting Primary care.
Participants Two general practitioners in the West Midlands who were associated with an unacceptably high mortality of patients during 1993-2000.
Main outcome measures Observed and expected number of deaths and deaths in nursing homes.
Results Preliminary discussions with the general practitioners highlighted deaths in nursing homes as a possible explanatory factor. No relation was found between the expected number of deaths and deaths in nursing homes in each year during 1993-2000 for either general practitioner. In contrast, the magnitude and shape of the curves of a cumulative sum plot for excess number of deaths (observed minus expected) in each year were closely mirrored by the magnitude and shape of the curves of the number of patients dying in nursing homes; and this was reflected in the high correlations (R2 = 0.87 and 0.89) between excess mortality and the number of deaths in nursing homes in each year for the general practitioners. These findings were supported by administrative data.
Conclusions The excessively high mortality associated with two general practitioners was credibly explained by a nursing home effect. General practitioners associated with high patient mortality, albeit after sophisticated statistical analysis, should not be labelled as having poor performance but instead should be considered as a signal meriting scientific investigation.
After the crimes of Dr Harold Shipman, the UK government set up an independent inquiry.1 2 Important issues for the inquiry were the suggestion that a statistical analysis of Shipman's data on patient mortality could have spotted his activities earlier and the recommendation that death rates in general practice should be routinely monitored.1 3–5
The Shipman inquiry commissioned an independent research team from Imperial College, London, to consider the monitoring of death rates in general practice. The team showed that such monitoring was feasible and tried its preferred statistical method of cumulative sum plots in a pilot study.4 6 This plot, which in essence is an accumulation of the standardised differences between observed and expected numbers of deaths, takes account of several sources of variation, including chance, patient case mix, seasonality, and systematic (but unknown) sources of variation. When the plot crosses a statistically predefined threshold, which is akin to a significant P value with allowance for multiple testing, it is said to have “signalled” an “unacceptably” high mortality that seemed to implicate quality of care. In the pilot study, involving a sample of 1009 general practitioners from five health authorities, cumulative sum plots highlighted 12 general practitioners associated with unacceptably high patient mortality who merited investigation.4 6 One of the general practitioners was Shipman. As for the rest, the UK Department of Health wrote to the appropriate health authority or primary care trust and requested an investigation of the high mortality. We investigated two general practitioners associated with high patient mortality who were in the West Midlands, our area of responsibility. We sought to identify a credible explanation for the associated high death rate.
We adopted the pyramid of investigation model (fig 1).3 This model checks five variables—data, patient case mix, structure, process of care, and carers. We began with a review of the data. The general practitioners (A and B) were made aware of, and fully cooperated with, the investigation.
Through the Shipman inquiry we obtained the raw data on mortality for the general practitioners and the analyses conducted on these data by the team from Imperial College, whose methods are fully described elsewhere.4 6 Briefly, their raw data consisted of a list of the deceased patients for each general practitioner, with personal details (date of birth, sex, postcode) and date, place, and cause of death. Their analyses included a count of observed and expected numbers of death for each year from 1993 to 2000 together with the cumulative sum plots. The expected numbers of deaths were determined using indirect standardisation adjusted for patients' age, with reference rates being derived from the relevant health authority. The cumulative sum plots were designed to detect general practitioners associated with a patient mortality of at least 4 standard deviations higher than the acceptable level. The threshold for signalling an alarm was set at 3, which was almost certain (> 99.9%) to indicate a true signal (fig 2, top). The cumulative sum plots for general practitioners A and B crossed the alarm threshold in 1996.
We used the codes for place of death in the raw data to determine the address of the place of death through the Office of National Statistics Communal Establishment File. The table shows the data obtained from the Shipman inquiry.
We undertook the generation and testing of an exploratory hypothesis using the raw and adjusted data. From these we sought specific items of data, analysis, or information that could be used to test the validity of the hypothesis.
Generation of hypothesis
After preliminary discussions with general practitioners A and B and the local lead for underperformance of general practitioners (AR), it was apparent that there had been no major dissolutions in practice, epidemics, or changes in boundaries in the areas served by the general practitioners during 1993-2000. It was known, however, that in 1996 (the year the alarm threshold was crossed) general practitioner A had taken on substantial nursing home commitments and that general practitioner B was practising in a catchment area with several nursing homes. We sought to test the “nursing home effect” as our preliminary working hypothesis.
Testing the hypothesis
We undertook several analyses. Firstly, we compared the excess (observed minus expected) mortality with the number of deaths in nursing homes for each general practitioner (fig 2, middle). The shape and magnitudes of these variables for both general practitioners were apparently similar, which suggested that the excess mortality was attributable largely to deaths in nursing homes. We tested the relation between the excess mortality and deaths in nursing homes for each general practitioner using an XY scatter plot and found high correlation coefficients of 0.89 (general practitioner A) and 0.87 (B) using a best fit least squares line not constrained through the origin (fig 2, bottom).
We also explored the relation between expected numbers of deaths and deaths in nursing homes (fig 2, middle). The expected numbers of deaths associated with general practitioner A seemed to be insensitive to the number of deaths in nursing homes; for general practitioner B, however, there seemed to be some relation, which on closer inspection was confounded by the substantial increase in the list (for example, in 1998 the list size doubled as did the expected number of deaths; (table).
Administrative records showed that throughout 1993-2000, general practitioner A's practice was responsible for two nursing homes and that A was responsible for care in both (1993-2000 for one, 1992-1998 for the other). During 1993-2000 general practitioner A would therefore have been the practice doctor who signed off the most certificates for deaths in the nursing homes. The relinquishment of one of the nursing homes in 1999 is associated with a substantial decrease in the percentage of deaths (66% in 1998 compared with 14% in 1999; (table).
Administrative records for general practitioner B's practice showed that in 1993 the practice was responsible for two nursing homes. In 1994 the practice took on a nursing home. In May 1995 the practice took on yet another nursing home, which was relinquished in October 1997. In 1998 the senior partner retired and general practitioner B took on his list (1514 patients in 1997 compared with 3137 patients in 1998; (table). Before 1997, general practitioner B had sole responsibility for one nursing home and from 1997 had responsibility for three. The percentage of patients who died in nursing homes increased from 40% in 1993 to 65%-72% in 1994 and 1995 when the practice took on an extra nursing home and decreased from 78% in 1997 when a nursing home was relinquished to 54% in 1998 (table).
Our visual analysis of a limited dataset from the Shipman inquiry supports a “nursing home effect” for the high mortality of patients associated with certain general practitioners. We believe that our findings are credible on the basis of a graphical model, which combined the results of local knowledge, administrative data, and reported findings that patients admitted to a nursing home are known to have high mortality.7 8
Apart from the two general practitioners we investigated, we are aware of six others who also signalled “unacceptably” high patient mortality through the Shipman inquiry and whose investigations have been reported (A Rixom, personal communication, 2004).9 All six were subsequently found to have a nursing home effect. The reports describe a complex investigation process, which included several strategies encompassing quantitative and qualitative data in combination with an independent review of case notes. It remains unclear as to how useful the review of case notes was in these instances, although the methodological issues as well as time and cost are considerable.9 10 Nevertheless, our findings along with those of others, suggest a need for a replication of our analysis for the other nine general practitioners who also signalled “unacceptably” high mortality, and that the model of adjustment for case mix on which the cumulative sum plots are based may need to be refined to accommodate adequately the nursing home effect. We also perceive a need to monitor mortality in nursing homes independently.
An important implication of our finding is caution in over-interpreting data adjusted for case mix, which has been described as the case mix adjustment fallacy.11 This fallacy begins with a deceptively simple equation that relates the variance in outcomes (mortality) to a combination of three sources—chance, patient case mix, and quality of care. Sophisticated statistical techniques are often used to account for chance, both in measured and, in this case, unmeasured (by taking account of over dispersion in observed deaths) patient case mix factors, with the residual unexplained variance being prematurely assigned to quality of care.4 We have shown that such reasoning, although seductive, should be diligently avoided if the process of monitoring is to remain a credible and positive contribution to improving quality of care.12 A general practitioner associated with high patient mortality, albeit after a sophisticated analysis, should not be labelled as having poor performance but instead should be considered as meriting proper scientific investigation for a credible cause. If a national monitoring system is implemented, the need for local knowledge and expertise in interpreting the data should not be underestimated.
What is already known on this topic
After statistically sophisticated analyses, the Shipman inquiry was notified of 12 general practitioners (one being Harold Shipman) with excessively high mortality
Several of the 11 general practitioners have been investigated, mostly using the costly and challenging method of case note review
What this study adds
A “novel” pyramid model of investigation was applied to two general practitioners associated with unacceptably high patient mortality
The excess mortality is adequately explained by taking account of the proportions of patients dying in nursing homes
We thank general practitioners A and B for their cooperation and support throughout this work, the staff involved with the Shipman inquiry, Paul Aylin (Imperial College, London) for providing the relevant datasets, and A Rixom, J Billet, N Kendall, and P Old for sharing their experience.
Contributors AR headed the investigation team. DP and HO carried out data validation checks and exploratory analysis. PM was part of the investigation team and provided guidance and support throughout and was our link with the Department of Health and the team at Imperial College, London. MAM provided a framework for the investigation, undertook analysis of the dataset from the Imperial College team, found the results shown, and wrote the first draft of the paper. AS provided guidance and support throughout. All authors contributed to the writing of the final paper. MAM will act as guarantor for the paper.
Competing interests MAM was an expert witness at the Shipman inquiry, where he discussed the monitoring of death rates associated with general practitioners.
Ethical approval Not required.