Observations Body Politic

Too much information

BMJ 2011; 343 doi: http://dx.doi.org/10.1136/bmj.d4341 (Published 12 July 2011) Cite this as: BMJ 2011;343:d4341
  1. Nigel Hawkes, freelance journalist
  1. nigel.hawkes1{at}btinternet.com

The facts don’t always speak for themselves, and the prime minister may come to regret thinking that they do

The UK prime minister, David Cameron, wants to liberate a tsunami of public data. In the interests of transparency, not a cough or splutter in national government, local government, schools, the courts, or, of course, the NHS, will go unrecorded and made available to everybody.

The plans, announced on 7 July, include releasing data ranging from the prescribing habits of individual general practices to the success rates of clinical teams in hospitals, not forgetting the complaints made about every hospital (BMJ 2011;343:d4415, doi:10.1136/bmj.d4415). It is hard to argue that more information can ever be bad, but it’s permissible to wonder whether undigested data are quite the blessing that Mr Cameron imagines.

In his memoir A Journey, the former prime minister Tony Blair makes few acknowledgments of error in his 10 years of office. But one thing he does regret doing is introducing legislation on freedom of information. He came to regard it as a curse, making government much harder. “You idiot. You naive, foolish, irresponsible nincompoop,” he writes. “There is really no description of stupidity, no matter how vivid, that is adequate. I quake at the imbecility of it.” He goes on, unstoppably, like a drunk grabbing at your lapel: “For political leaders, it’s like saying to someone who is hitting you over the head with a stick, ‘Hey, try this instead,’ and handing them a mallet.”

So I think we can conclude that he changed his mind about throwing open the windows and letting the fresh air of transparency wash over the processes of government. His argument is that, in the formulation of policy, frank conversations are needed that may be inhibited by the risk that they will be later published and selectively quoted. Anybody who has ever run anything, from a major company to a cake stall at the Women’s Institute, can see the force of that.

The arguments over Mr Cameron’s tsunami are slightly different. In another life I serve (part time) as director of a campaign group, Straight Statistics, that seeks out misuse or misrepresentation of statistics and publishes them on a website. The statistical community, it is fair to say, is divided about the open government initiatives of the previous and the present government.

On one side stand the guardians of statistical purity who believe that data are dangerous without metadata. You cannot understand any statistics without knowing how they were gathered, by what techniques and by whom, and what their limitations are. Raw data without adequate statistical commentary can lead to argument and misunderstanding, muddying the waters of debate rather than clarifying them. Careful rules have been formulated, and are imposed, so that government departments do not simply toss out statistics without proper commentary. If they do, they are reprimanded.

On the other side of the divide are what might be called the free marketeers of data, who include the Cabinet Office minister, Francis Maude. They believe that even though statistics may be misinterpreted from time to time, the free play of opinion will ensure that truth prevails. They regard statistical commentary as a form of nannying. Haven’t people got the intelligence to make their own judgments on the basis of the facts? And won’t the free flow of data enable clever people to make use of it in diverse ways that will enrich understanding and expose error?

Both arguments are strong, but I lean to the first. It applies with even greater force to data about the performance of hospitals and individual doctors. Without some effort at risk stratification, death rates for individual surgeons or surgical teams are meaningless. The danger is that publishing them will drive surgeons into risk averse behaviour and that some patients will be denied treatment.

More than a decade of work has gone into formulating an effective measure even at hospital level, and we are still not quite there. Last November the Department of Health for England announced that its expert committee had devised its own measure, the summary hospital level mortality indicator (SHMI), which would be published from this April (BMJ 2010;341:c6298, doi:10.1136/bmj.c6298). Now April has come and gone and we still do not have SHMIs. I am told that they may start appearing in October.

The deliberations of the expert committee showed that even today not everybody is convinced of the value of such an indicator. It is open to manipulation, as Brian Jarman, the originator of a very similar measure called hospital standardised mortality ratios, recently explained to the inquiry into excess deaths at Mid Staffordshire NHS Foundation Trust. He said that Mid Staffordshire was one of three trusts in the West Midlands that “gamed” mortality indicators by increasing the numbers of patients coded as palliative care. During 2008 the proportion of patients so coded had risen to 30% at Mid Staffordshire, to 40% George Eliot Hospital NHS Trust, Nuneaton, and to 78% at Walsall Hospitals NHS Trust. “The only way you could get dramatic changes like that would be if the three trusts became terminal care hospitals overnight,” Sir Brian said.

The effect was to increase the number of expected deaths and reduce the mortality ratio, calculated from a comparison of expected to actual deaths, as I reported in April 2010 (BMJ 2010;340:c2153, doi:10.1136/bmj.c2153). Sir Brian told the Health Service Journal that the increase in palliative care coding is why the annual improvement in mortality rates in England seems to have overtaken those in countries such as the Netherlands and Germany.

My point is this: if a measure covering an entire hospital can be manipulated so easily, how are we to ensure that measures of individual surgical groups or doctors are not similarly manipulated? The facts speak for themselves, say the proponents of open data. But they don’t. Circumstances vary so widely that mortality rates need to be corrected for risk, and the techniques for doing so are imperfect and open to abuse. Mr Cameron may come to regret throwing open the door, though he is unlikely to do so in such colourful language as did Mr Blair.

Notes

Cite this as: BMJ 2011;343:d4341