Rapid responses are electronic comments to the editor. They enable our users
to debate issues raised in articles published on bmj.com. A rapid response
is first posted online. If you need the URL (web address) of an individual
response, simply click on the response headline and copy the URL from the
browser window. A proportion of responses will, after editing, be published
online and in the print journal as letters, which are indexed in PubMed.
Rapid responses are not indexed in PubMed and they are not journal articles.
The BMJ reserves the right to remove responses which are being
wilfully misrepresented as published articles or when it is brought to our
attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not
including references and author details. We will no longer post responses
that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Using Scattergrams to replace rumours about GP Practices
We, like Keogh, take the view that opening up huge national data
bases has the potential to drive up quality of patient care. However,
Buckman has a very valid concern when he says, "outcome measures are
dificult to interpret". Nevertheless we believe that the answer to the
the important question, "Is it likely that practice X is not offering a
quality service?" can be answered simply and rigorously.
10 years ago the most difficult part of any performance evaluation
was overcoming the paucity of available data. These days the difficulty
is to avoid drowning in a torrent of data. A torent that will inevitable
become a deluge when this "open data programme" starts. Consider this:
The UKs 2009/10 QoF databases contain data on >10K practices; and each
is assesssed on >70 clinical performance criteria. In the past how has
such a mountain of data been turned into useful information? Well the
answer is that generally it hasn't! Ingenius devices such as
"Dashboards" have been contrived but it is clear that their utility not
matched the enthusiasm of their promoters. And of course we have the
dreaded, "simplistic league tables" which Buckman warns against.
We suggest that, when a myriad of data need reviewing on each
practice, the use of "achievement scattergrams" be considered. Five
scattergrams (which relate to performance of 3 practices with a partner
who has contributed to this weeks BMJ, and two anonymysed practices) are
shown in figure 1 . Data relates to 62 clinical QOF criteria. We have
experimented and developed this tool over a 5 year period and have reason
to believe that this practice performance of this PCT's has improved as a
result.
Scattergrams are easy to prepare, and conceptualy easy to intepret.
For instance it is easy to obtain QOF data on all of the 10K practices in
the UK and calculate the median performance value on each of 62 QOF
criteria. These minimum values are then plotted on a graph as the sloping
achievement lines shown in Figure 1. 62 data scatter points which
represent the achievement of any practice can now be plotted on the same
graph. The distribution of achievemnt scores of all UK practices is shown
in Fig. 1A. Graph 1A shows that, as our intuition suggests, most practices
achieve ~31 points above the line and ~31 below the line. However the
range is very wide and there is no obvious "cut-off" that can be used to
justify, "pressing the red button". However, we are certain that whilst
no reader will have concern about achievments of practices 1B and 1C (54
and 41 achievement points), we are equally certain that all readers would
identify practices 1E and 1F (14 achievement points) as meriting further
investigation.
50 years ago the Japanese revolutionised the meaning of high quality
by adopting the philosophy that the performance of a high quality system
is , "On-target, minimum-variation". High variation patterns are easily
detected by scattergrams. For instance, scatter points of practices 1B
and 1C are clearly tightly clustered (minimal-variation); on the other
hand scatter points of practices 1E and 1F clearly show high variation and
as such are unlikely to represent a "high quality system. Amongst the 10k
practices across the UK there are several hundred whose achievement
scattergrams show enormous - sometimes blunderbus - variation. We have
chosen not to display these but worried readers can check their own
practices achievement scattergram by going to the web application at http://www.realtool.co.uk
So to answer the question, "Is it likely that practice X is not
offering a quality service?" the answer is, "If practice X's scattergram
shows wide variation there is some cause for concern however if the
scattergram shows wide variation AND a low achievement score, there is
great cause for concern"
Re: Primary care datasets will replace "rumours" about GPs' performance : BMJ 2011; 343:d4415 (Published 12 July 2011)
Using Scattergrams to replace rumours about GP Practices
We, like Keogh, take the view that opening up huge national data
bases has the potential to drive up quality of patient care. However,
Buckman has a very valid concern when he says, "outcome measures are
dificult to interpret". Nevertheless we believe that the answer to the
the important question, "Is it likely that practice X is not offering a
quality service?" can be answered simply and rigorously.
10 years ago the most difficult part of any performance evaluation
was overcoming the paucity of available data. These days the difficulty
is to avoid drowning in a torrent of data. A torent that will inevitable
become a deluge when this "open data programme" starts. Consider this:
The UKs 2009/10 QoF databases contain data on >10K practices; and each
is assesssed on >70 clinical performance criteria. In the past how has
such a mountain of data been turned into useful information? Well the
answer is that generally it hasn't! Ingenius devices such as
"Dashboards" have been contrived but it is clear that their utility not
matched the enthusiasm of their promoters. And of course we have the
dreaded, "simplistic league tables" which Buckman warns against.
We suggest that, when a myriad of data need reviewing on each
practice, the use of "achievement scattergrams" be considered. Five
scattergrams (which relate to performance of 3 practices with a partner
who has contributed to this weeks BMJ, and two anonymysed practices) are
shown in figure 1 . Data relates to 62 clinical QOF criteria. We have
experimented and developed this tool over a 5 year period and have reason
to believe that this practice performance of this PCT's has improved as a
result.
Scattergrams are easy to prepare, and conceptualy easy to intepret.
For instance it is easy to obtain QOF data on all of the 10K practices in
the UK and calculate the median performance value on each of 62 QOF
criteria. These minimum values are then plotted on a graph as the sloping
achievement lines shown in Figure 1. 62 data scatter points which
represent the achievement of any practice can now be plotted on the same
graph. The distribution of achievemnt scores of all UK practices is shown
in Fig. 1A. Graph 1A shows that, as our intuition suggests, most practices
achieve ~31 points above the line and ~31 below the line. However the
range is very wide and there is no obvious "cut-off" that can be used to
justify, "pressing the red button". However, we are certain that whilst
no reader will have concern about achievments of practices 1B and 1C (54
and 41 achievement points), we are equally certain that all readers would
identify practices 1E and 1F (14 achievement points) as meriting further
investigation.
50 years ago the Japanese revolutionised the meaning of high quality
by adopting the philosophy that the performance of a high quality system
is , "On-target, minimum-variation". High variation patterns are easily
detected by scattergrams. For instance, scatter points of practices 1B
and 1C are clearly tightly clustered (minimal-variation); on the other
hand scatter points of practices 1E and 1F clearly show high variation and
as such are unlikely to represent a "high quality system. Amongst the 10k
practices across the UK there are several hundred whose achievement
scattergrams show enormous - sometimes blunderbus - variation. We have
chosen not to display these but worried readers can check their own
practices achievement scattergram by going to the web application at
http://www.realtool.co.uk
So to answer the question, "Is it likely that practice X is not
offering a quality service?" the answer is, "If practice X's scattergram
shows wide variation there is some cause for concern however if the
scattergram shows wide variation AND a low achievement score, there is
great cause for concern"
Figure 1 - Example of Scattergrams
References
http://www.realtool.co.uk/Downloads/How to prepare median performance
scattergrams - dataset.xls
http://www.realtool.co.uk/perfprepared.html
http://www.realtool.co.uk/perfidentity.html
http://www.ehsmeds.co.uk/main-meds-news-title-be-inserted-here-2
Competing interests: No competing interests