The BMJ paper and seven day services

Where did the 6000 figure come from?

Immediately after Hunt’s speech, the source of the figures, particularly the 6000 excess deaths, was questioned. There did not, however, seem to be a clear answer. BBC journalist Jane Dreaper was told by the Department of Health that the source was a paper that would soon be published in The BMJ.2 On 16 July she tweeted, “Answer now from @DeptHealthPress—apparently it relates to analysis of 13/14 HES [Hospital Episode Statistics] data being published in BMJ soon.”

Meanwhile Laura O’Brien, a writer for fact checking charity Full Fact, was told by the health department on the same day that the figure had come from Bruce Keogh, NHS England’s medical director.3 Concerned that Hunt had “used unpublished figures from NHS England,” O’Brien emailed the UK Statistics Authority that day to complain.4

On 24 July, Ed Humpherson, UK Statistics Authority director general for regulation, wrote to Mark Svenson, head of analytical services at NHS England, summarising their earlier conversation.5 Humpherson wrote that Svenson had explained that the figures quoted by Jeremy Hunt came from “medical research, which you expect to be published by the researchers in the British Medical Journal in due course.”

An email from NHS England on the day that Hunt gave his speech seemed to confirm this view. The email, released under a freedom of information request, was sent to Domenico Pagano, Nick Freemantle, and Keogh, all authors of The BMJ paper, by NHS England’s Seven Day Services Forum project manager, Deborah Williams.6

In it she said, “We were challenged to cite the source of the 6000 figure and attempted to offer up the most bland statement possible, that would neither confirm nor contradict the [health secretary’s] statement—he had already given his speech and in follow up interviews was saying that this was a number endorsed by Bruce [Keogh].

“As you know, the ‘estimate’ was made by Deloitte who had been sighted on [sic] your forthcoming paper in the course of their modelling work. I’m copying our Head of Media to ask what use has already been made of the line and how we might construct, with your help, a line that supersedes and therefore reduces any risk to your publication.”

However, in his reply to this email, sent the same day, Pagano told Williams that these messages from NHS England were inaccurate. He alluded to a “proposed press statement” from NHS England, which was sent to Freemantle on 16 July who then forwarded it to Pagano, which said that the 6000 figure was “based on an analysis of 2013/14 HES data that is due to be published in the BMJ shortly.”

Pagano said, “Unfortunately this is not the case. Our work does not refer to 6000 deaths as a result of increased mortality risk associated with weekend hospital admission.”

On 20 July, four days after Hunt’s speech, Keogh and Simon Stevens, chief executive of NHS England, gave evidence to the Commons Health Select Committee on the current performance of the NHS.7 During this meeting Keogh repeated the 15% figure, but he did not mention the 6000 excess deaths.8

During his evidence, Keogh explained that in 2012 he had worked with a University Hospital Birmingham research group, which included Pagano and Freemantle, to look at hospital episode statistics for the year 2009-10. This followed work by Dr Foster that had shown higher mortality in some hospitals at weekends. The results, he said, were published in February 2012 in a paper in the Journal of the Royal Society of Medicine.9

NHS England has confirmed that the 6000 figure was derived from this 2012 paper, and that it shared this figure with the Department of Health. It said that the figure had been discussed widely within the organisation, and it had been calculated both internally within NHS England and by Deloitte as part of its work for NHS England on seven day services.

“The 6000 figure which we shared with the Department of Health was not in the 2012 study but it was derived from it,” Keogh told The BMJ. “It is a shame there has been confusion because that has detracted from a very important debate about ambition and improvement.”

On 13 August, the Department of Health published an explanation of how the 6000 figure could be derived from the 2012 paper.10 It said, “The estimated excess in-hospital deaths associated with admission on Saturday or Sunday compared with Wednesday is 5745.”

NHS England has told The BMJ that the health department’s explanation of the source of the 6000 figure wasn’t published until August because key officials who could confirm it were unavailable.

So according to NHS England the 6000 figure was derived from the 2012 paper, both by NHS England and by Deloitte during its work for NHS England. NHS England said it was its figure, and not Deloitte’s, that was shared with the Department of Health. It remains unclear, however, why two versions of the same figure were calculated.

Freemantle, who was not involved in the calculation of the figure or with sharing it, told The BMJ he had “no idea” where it had come from.

Did the Department of Health commission the new analysis?

Keogh explained the provenance of the 2012 paper to the Health Select Committee on 20 July. “We looked to see whether the observation that the mortality at the weekend was indeed translatable to the whole of the NHS,” Keogh said, “The end result was that there were some interesting findings.”

Keogh told the committee that the paper showed that mortality was 11% higher for patients admitted on a Saturday and 16% higher for patients admitted on a Sunday when compared with a Wednesday.

He also told the committee that Stevens had subsequently asked him to update the 2012 paper. “Simon asked me to...go back and look at more recent figures to see whether the mortality still prevails, and we are in the process of publishing this—or hoping to,” Keogh said. “The high-level findings from that are, that on a Saturday now the mortality, as Simon said, is 10%, as opposed to 11%. On a Sunday, it is elevated by 15% as opposed to 16%.”

NHS England has told The BMJ that it asked the University Hospital Birmingham research group to update its analysis using 2013-14 data in February 2015. This request followed a meeting in November 2014, at which the management consultancy Deloitte was invited to update the senior management team of NHS England’s medical directorate on work that NHS England had commissioned the company to carry out on seven day services. The same update was then presented to the senior leadership of NHS England in February 2015, when the further information and analysis were requested.

On 5 September 2015, The BMJ published an analysis article by Keogh and colleagues entitled, “Increased mortality associated with weekend hospital admission: a case for expanded seven day services?”11

It had been submitted on 22 May 2015 and provisionally accepted on 22 July. It was revised and resubmitted on 9 August and accepted on 18 August. The article then underwent content and technical editing before publication.

The authors describe The BMJ article as an update of their 2012 paper. “Because six years have elapsed since our last assessment of weekend mortality we have updated our analysis using data from NHS English hospitals and on related deaths in 2013-14,” they state.

The study found that “the relative risk of death within 30 days compared with admission on a Wednesday was increased by 2% for admission on a Friday, 10% for admission on Saturday, 15% for admission on a Sunday, and 5% for admission on a Monday.”

The article does not refer to 6000 excess deaths, as quoted by Hunt. What it does say is that 11 000 more people died each year “within 30 days of admission to hospital on Friday, Saturday, Sunday, or Monday compared with other days of the week.”

The authors said it was “not possible to ascertain the extent to which these excess deaths may be preventable; to assume that they are avoidable would be rash and misleading.”

The article does not state that it was commissioned by NHS England’s senior management team, but Fiona Godlee, editor in chief of The BMJ, says the authors would not have been required to include this information. “This information could have been included, but the fact that it was not does not strike me as worrying,” Godlee says, “The fact that one of the authors was NHS England’s medical director would anyway indicate the origin of the idea for the reanalysis.”

Were data from the new analysis shared with the DH?

Both NHS England and the authors of the article have stated that it was not shared with anyone in the Department of Health before publication. Freemantle told The BMJ, “I’ve been asked a number of times about whether the secretary of state for health had commissioned [the results] and whether I had shared them with him and whether I had briefed him on them.”

“He didn’t commission them. I have never spoken to him [Hunt] in my life. I had no contact with him or anyone in his office in between us undertaking the analysis and publishing the results.”

The BBC has reported that NHS England’s freedom of information officer said in October last year, “Sir Bruce Keogh did not discuss the study’s findings with anyone at the Department of Health at any point before or after the publication of this article.”6

However NHS England subsequently told the BBC that this statement is “clearly a mistake, and absurd.”

It says that top level findings from the article, including the finding that mortality was 15% higher on a Sunday, were shared by Keogh with policy colleagues at NHS England, the Department of Health, health unions, royal colleges, the Health Select Committee, and Deloitte before publication.

Despite hearing the top level findings, a spokesperson from Deloitte confirmed in an email sent to NHS England on 11 January 2016, that it “did not see the Freemantle paper, final or draft, prior to publication.”

NHS England said that Keogh also shared the top level findings with the Review Body on Doctors’ and Dentists’ Remuneration (DDRB) in March 2015. Keogh had been asked to give evidence to the DDRB on the case for improving seven day services, NHS England said.12

Keogh defends the decision to share the 2015 article’s top level findings before it was published. He argues that that it is vital to the development of policy that research is shared.

“I am keen to promote transparent evidence based policy making. This debate has exposed a tension,” Keogh says. “If the suggestion is that we shouldn’t share emerging information with our own policy teams while we wait for the information to go through a peer review process then delay means it would become more difficult to publish in journals. This will stifle informed debate, and that becomes a very sad state of affairs.”

Freemantle confirmed to The BMJ that he also shared top level results before the article was published. “I presented part of them at a methodological conference that was held in London in June [2015]. They were buried in among some other pieces of work,” he says. “But the results are on the slides and described quite clearly as an update of our previous work.”

Tony Delamothe was the deputy editor at The BMJ who handled the 2015 article. He says it is not wrong for authors to share the top line findings of a paper before it is published. “People share findings: they share them with the subjects [of the research], sometimes they share them with the drug company whose drug they have been looking at. There’s nothing that says you can’t talk to anyone about this paper and if you do, you have to say in the paper that you have.”

Nor was it unusual, Delamothe says, for researchers to share top line findings from a paper before they are published. He also says the fact that Hunt mentioned figures from the article in a speech before it was published would not have broken any BMJ embargo.

“If somebody says, ‘I’ve got this paper and its being considered by The BMJ and we want to mention the headline finding at a conference,’ we would say, ‘That’s absolutely fine but you can’t hand out chapter and verse to people at the conference,’” Delamothe says. “‘You can give the headline finding but none of that detail, because that’s why you’ve sent the paper to us.’ Our instructions to authors makes that clear. In no way did [Hunt’s speech] break a BMJ embargo on the paper.”

Was The BMJ’s peer review process and decision to publish influenced by Hunt’s speech?

In Pagano’s email to NHS England, released under freedom of information, he raises the concern that attributing the 6000 figure to the 2015 paper would not only be inaccurate but could also interfere with the peer review process. “It will be inaccurate and counterproductive to quote that our analysis is due to be published in The BMJ shortly, as this is not the case and may seem to interfere with the peer review process,” he wrote.

Freemantle told The BMJ that his main concern after Hunt’s speech was that the journal would decide not to publish the article at all, since it had not been formally accepted on 16 July. “I thought there was a very good chance that The BMJ would say, ‘Well, don’t go anywhere near this paper.’ At that point I thought it would prejudice the editorial process against us,” Freemantle says.

Delamothe, however, says it was highly unlikely that Hunt’s speech had any effect on the editorial process. He says that the two peer reviewers who looked at the paper for The BMJ received it and returned their views before Hunt gave his speech.

“I sent it to two referees. Massimo Gallerani, who works in Ferrara, Italy; I sent it out for his opinion on 30 June and he returned it on 8 July. The second referee was Dylan de Lange, who works in Utrecht in the Netherlands. I sent it to him on 30 June and he sent it back on 30 June,” Delamothe says.

The paper was then sent to a statistician, Gary Collins, on 7 July. The BMJ team handling the paper decided at a meeting on 9 July that they could consider the paper for publication once they had heard back from Collins. He submitted his opinion on 23 July, seven days after Hunt’s speech. Delamothe says that although there may have been potential for Collins’s view to have been influenced by Hunt’s speech, this was highly unlikely.

“I would think it’s unlikely because we don’t really employ statisticians to get swayed by what they read in the media. It doesn’t seem likely that some politician who’s here today and gone tomorrow is going to influence our statistician at a high powered unit in Oxford,” Delamothe says.

By the time Hunt made his speech on 16 July, Delamothe says that he was the only person dealing with the paper. He provisionally accepted the paper, provided the authors would revise it in line with the reviewers’ comments, on 22 July. “Was I swayed by media coverage of Hunt’s speech? I really don’t think I was,” he says.

Several commentators have asked to see the peer review reports for the paper. Delamothe explains that, in order to publish these, The BMJ needed to get the approval of the two peer reviewers and the statistician. This is because, in contrast to its policy for research articles, the journal doesn’t routinely publish the prepublication history for analysis articles. “It’s taken a while to line up these three people to say yes. If these guys had come back with a yes the day that I asked them, it would have gone up instantaneously,” Delamothe says. Two of the peer review reports have now been published and the third is awaiting permission from the reviewer.