Data too important to share: do those who control the data control the message?BMJ 2016; 352 doi: https://doi.org/10.1136/bmj.i1027 (Published 02 March 2016) Cite this as: BMJ 2016;352:i1027
- Peter Doshi, associate editor, The BMJ
Yale cardiologist Harlan Krumholz has scored some notable successes in his advocacy of liberalising access to clinical trial data. Last year he embarked on a collaboration with one of the world’s largest drug companies—Johnson and Johnson—to open its vaults and begin sharing data with independent researchers.1 This feat followed a highly publicised reanalysis of a Medtronic spinal treatment that had come under fire for being unsafe.2 3
But Krumholz, always excited about the idea of applying his Yale open data access (YODA) model to new situations, admits defeat in his latest attempt at uncovering raw trial data.
It’s an important case because it flips the usual narrative of industry refusing to share data with academics. It involves a large randomised controlled trial comparing saline and hydroxyethyl starch (HES) to replace lost blood volume in critically ill patients (the CHEST trial) and a group of academics that refuse to share their data with an industry sponsor.
At the centre is John Myburgh, a well known figure in intensive care medicine and the principal investigator of the CHEST trial, which included 7000 patients.4 In 2012, Myburgh and colleagues from the George Institute for Global Health in Australia published their findings in the New England Journal of Medicine,5 raising safety concerns that helped seal the fate of starch solutions across much of the world.
In 2013 the US Food and Drug Administration added boxed warnings to HES products noting an increased risk of death and need for renal replacement therapy.6 In Europe the European Medicines Agency (EMA) recommended suspension of infusion solutions containing HES and the United Kingdom recalled the drugs.7 8 However, at the end of 2013, the EMA changed its recommendation, concluding that although HES solutions were contraindicated in critically ill patients or patients with sepsis or burns, they could be used to treat acute blood loss when treatment with crystalloids alone is insufficient.
Fresenius Kabi, one of the major manufacturers of HES products and a major funder of the CHEST trial (€3.5m; £2.7m; $3.8m), questions the trial’s reporting in the NEJM and wants to verify the data. However, under the terms of its contract with the George Institute, it signed away all rights to the data. This decision now haunts the German manufacturer.
“We have many specific concerns about the publication, all of which could be resolved if we were in a position to verify the reported results against raw data of the study,” Fresenius Kabi senior vice president, Hrishikesh Kulkarni, told The BMJ. The company says sales of HES dropped in 2013 but have recovered over the past two years.
“This case exemplifies the broader cultural shift in medicine,” commented physician author and transparency advocate Ben Goldacre, whom Fresenius Kabi contacted for help. “The era of researchers or sponsors feeling entitled to conduct secret analyses in private, without any wider scrutiny from the academic and clinical community, is coming to an end.” He added: “The researchers should hand these data over, and if they want to be taken seriously, the sponsors [Fresenius Kabi] should set out their protocol for analysing it before they receive the files.”
After Myburgh turned down direct requests for data from Fresenius Kabi, the company approached The BMJ and is calling for an independent reanalysis. The BMJ in turn suggested that the company take its case to Krumholz for application under the YODA model. YODA agreed to serve as an independent intermediary between the CHEST investigators and the yet to be determined reanalysis team. Fresenius Kabi agreed to fund Yale’s efforts but otherwise not influence the process. As a first step, Krumholz approached lead CHEST investigator Myburgh to discuss releasing the data to Yale.
The answer is no
But after a flurry of emails, the CHEST investigators formally refused YODA’s request.
“The conclusion is that we cannot provide data for an analysis funded or otherwise supported by Fresenius, given their conflict of interest and repeated efforts to discredit the CHEST trial in an effort to protect their commercial interests. While we do not suggest your analysis would be biased by support from Fresenius, we cannot agree to provide data in these circumstances,” wrote Vlado Perkovic, executive director of the George Institute.
After denying access to the CHEST trial, Perkovic wrote that they “would consider a pilot process involving a less commercially-sensitive trial.”
“I do recognize that there is some disagreement regarding the trial between you and the sponsor. I don’t even know the details and did not want to know it since it was not relevant to what we were trying to do,” Krumholz wrote in reply. “The funding by Fresenius provides us the support to distribute the data—but they have no involvement in any aspect of the release. It seems, in the end, that you do not trust our independence. I am not sure what else I could do in that regard—and, as I said, our reputation is one of being tough on industry, not their pawns.”
Myburgh says his relationship with Fresenius quickly turned sour once the study results were not as the company had hoped. “The early days were superb,” he told The BMJ in an interview. “They were excellent collaborators.”
But by the time YODA came to intervene, suspicions of foul play abounded.
“We want restrictions on who could do the analyses”
“The main concern we had was that Fresenius was involved in the process,” Myburgh explained. He said there was never any question of Krumholz’s independence or credentials. Rather, it was a “concern that this was a way for Fresenius to get the data once they were in the public domain. We want restrictions on who could do the analyses.”
Under the YODA model Krumholz proposed, the data would be reanalysed by independent parties before being made more broadly available.
“We have no issue with the concept of data sharing,” Myburgh said. “The concerns we have come down to the people with ulterior motives which contradict or do not adhere to the scientific principles we adhere to. That’s the danger.”
Myburgh described himself as an impartial scientist, in contrast to those who have challenged his study. “I’ve heard some of the protagonists of starch. Senior figures wanted to make a point. We do research to answer a question. They do analyses to prove a point.”
Dissecting the study
Myburgh is confident about the conclusions that can be drawn from his trial.
“Trials,” he told us, are done “primarily to answer one question.” The CHEST trial, he said, “answered one question for which we have a definitive answer”—the effect on the primary outcome of 90 day all cause mortality.
“We showed that there was a 6% relative increase in the risk of death associated with the use of hydroxyethyl starch compared to saline, although this result was not statistically significant” (95% confidence interval −4% to 18%, P=0.26).
Myburgh said that CHEST was “specifically designed” to look at two endpoints: death and use of renal replacement therapy. And unlike mortality, the investigators reported a significant (P=0.04) increase in use of renal replacement therapy in the HES group.
The NEJM paper, however, gives 12 other secondary outcomes, although only renal replacement therapy is mentioned in the abstract’s conclusion.
Myburgh argued that renal replacement therapy is a clinical outcome and more relevant than other renal outcomes reported in the paper. Yet the other renal outcomes used a validated measure, and two were statistically significant, both in favour of HES. By contrast, no criteria were set for the initiation of renal replacement therapy in the CHEST study.
Furthermore, some have questioned the robustness of the finding, pointing to the NEJM paper’s supplementary appendix in which an “adjusted” analysis reports the P value as 0.05.9
Myburgh was quick to shoot down the importance of statistical significance, but another CHEST study investigator who was not a coauthor of the NEJM article told The BMJ on condition of anonymity: “I was uncomfortable with the way the renal complications were interpreted [in the NEJM paper]. We found two opposite effects: more use of renal replacement therapy in the HES group but more renal risk and injury according to the RIFLE criteria in the saline group.”
The conflicting renal results seem to reinforce the importance of access to raw data. “We don’t know what has caused this discrepancy, and we shouldn’t jump to the most convenient conclusion,” the investigator said.
Fresenius’s concerns are broader, and the company is reluctant to trust any of the study results because of problems it claims to have discovered in the CHEST investigators’ handling of adverse event data. In the NEJM abstract Myburgh and colleagues state that HES is associated with “significantly more adverse events (5.3% v 2.8%, P<0.001).” But Kulkarni told The BMJ: “It must have been derived by shifting between numerators, a number of patients from saline over to the HES group.” A correct analysis, of the intention to treat population, the company suggests, would not show a significant difference.
The company first wrote to the NEJM in 2013 to raise its concerns over adverse events, which were mainly pruritus. But Kulkarni says the journal responded that “the information contained in the CHEST study as published by us including supplemental material is accurately reported. Therefore we do not feel that any change is needed in our published material.” Fresenius’s letter went unpublished.
The BMJ contacted the NEJM for comment, but it responded: “Our editorial process is confidential, so we can neither confirm nor deny details related to anything that we have received and reviewed; we can only comment on the content that is published in the pages of NEJM.”
A document obtained by The BMJ under a freedom of information request shows that the number of adverse events that occurred in patients randomised to saline but counted in the HES group was 29 (see supplementary data on thebmj.com). According to a footnote in the NEJM paper, adverse events were counted under the HES tally irrespective of which study drug the patient was randomised to receive. Adverse events in patients randomised to saline but who received HES before randomisation were counted under the HES tally. The reverse, however, did not happen: adverse events in patients randomised to HES but who received saline before randomisation were not shifted to the saline tally.
In late 2015, Fresenius reiterated its concerns to the NEJM about the reporting of adverse events. “The reported P value is wrong,” it said in its letter. NEJM sought Myburgh’s opinion, who responded that the paper had reported incorrect denominators owing to a “typographical transcription error.” Furnishing a new P value of 0.006, Myburgh wrote that the error does not change conclusions. The NEJM told Fresenius: “We do not find that there has been any breach of scientific protocol and thus consider the matter closed.”
But Myburgh’s letter does not explain why the CHEST investigators attributed adverse events to HES for patients who received HES before or after randomisation but did not do the same for saline.
Neither the published protocol nor the statistical analysis plan for the CHEST study discuss this particular approach to analysing adverse events. In an interview, Myburgh indicated that the NEJM required the analysis “for better reporting.” Originally, he says, he had stuck it in a supplementary appendix.
Even regulators must trust the messenger
The case of HES solutions highlights the degree to which current scientific publishing practices and regulatory decisions are based on blind trust and strengthens the call for a shift to open data.
When regulators issued public warnings about HES products, they did so without any access to the CHEST trial’s underlying data. A spokesperson for the Australian Therapeutic Goods Administration (TGA) told The BMJ: “The TGA does not possess the Clinical Study Report or individual patient data from the CHEST study. The TGA has not requested the CHEST study data.”
The BMJ can also confirm that the FDA did not have raw data for at least two of the three randomised trials underlying its 2013 warnings over HES solutions6: CHEST and the 6S Study, which was the only randomised trial to report a significant increase in risk of mortality from HES (box 1).
Nothing will be shared, including the protocol
It is not just the CHEST study data that are out of public sight. The trial’s original protocol is also not available, even though NEJM has been posting them electronically alongside trials since 2011.10 Instead of the protocol, the online appendix of the CHEST trial directs readers to a 2011 publication of the protocol and a 2012 publication of the statistical analysis plan, both published well after the trial began enrolling patients. As a result, the reported study design, aims, and outcome definitions cannot be compared against those at the start of the study.
Myburgh refused a request by The BMJ for these documents: “With regard to the release non-public documents specifically related to the CHEST study, we do not consider that there is a convincing case that such release is in the best interests of patients, investigators, and the sponsor alike.”
The CHEST study investigators also refused to disclose a copy of a sample patient consent form, which Myburgh says includes provisions that “specifically preclude” releasing data. However, a copy of the patient consent form seen by The BMJ simply promises to keep “identifiable information” confidential, an almost universal provision in clinical trials, and as such seems fully consistent with the release of de-identified data (see supplementary data on thebmj.com).
In secrecy we trust
As well as the regulatory decisions being made without full data, two major systematic reviews of HES have been conducted based on journal publications alone.
Ian Roberts is coauthor of a Cochrane review on colloids versus crystalloids for fluid resuscitation in critically ill patients. His review concluded that colloids (including HES) did not increase the chances of survival and starches, in particular, “might increase the risk of death.”
“I see no basis for infusing into humans boiled up potatoes without evidence that the treatment offers benefit to the patients,” Roberts said in an interview, referring to starches.
“I do have a view on starch solutions. I think they should be banned.” He pointed to “no evidence of clinical benefit … if anything, the evidence suggests a harm.” He described Fresenius’s efforts as one of protecting its financial interests, not a pursuit of the truth.
But on the question of whether the data should be shared, Roberts was unequivocal. “I don’t think it should be John Myburgh’s decision. If a public funder pays for a clinical trial, they have a public responsibility to make the data available.” CHEST’s other major funder ($1.6m; £1.1m; €1.4m) was the Australian National Health and Medical Research Council.
“I think the CHEST investigators are shooting themselves in the foot. They have valid concerns, but it would be better if they made [the data] open.”
Another major systematic review on the topic, published by Ryan Zarychanski and colleagues in JAMA, came to similar conclusions as the Cochrane review. But this review also did not look any further than the NEJM publication of CHEST. “Data access is important, but must be done in a deliberate and thoughtful manner,” Zarychanski says. “Secondary analyses should be thoughtful, be placed into appropriate context, and should be with engagement of the primary investigators. Those with conflicts of interest (and deep pockets) have real potential to produce biased secondary analysis in the absence of such process.”
Mirror images—the story of the 6S trial
The data vacuum in which all regulatory decisions and systematic reviews are taking place extends beyond CHEST and also includes the 6S study.12 Like CHEST, the 6S trial compared a HES product versus a crystalloid (Ringer’s acetate). Like CHEST, 6S was published in the NEJM, and also in 2012. Like CHEST, the 6S investigators concluded HES inferior, and reported a statistically significant increase in risk of death on HES. And like CHEST, the manufacturer of the tested HES solution—B Braun—has concerns about 6S, but the academic investigators have thus far refused access.
B Braun senior vice president medical scientific affairs, Ute Brauer, told The BMJ that her company thought that there were “open questions” about the study conduct and interpretation “that could potentially be solved when the full data set is made available.”
6S lead investigator Anders Perner sees it differently. “As I remember it B Braun asked for additional analyses, which were not part of the protocol or statistical analysis plan (SAP), so we declined to do them . . . B Braun expected to have a more detailed report with more data, which we again declined ... We find it scientifically sound to report the analyses outlined in the protocol and the SAP and this was done in the NEJM paper.”
Thus far, he says, raw data have been shared only internally. “In general we support sharing of data, but models have to be developed ensuring protection of patient’s anonymity and investigator’s intellectual rights. In addition, we’re concerned about biased and less valid post hoc analyses, eg, data fishing.” He said that his group intends to publish the final dataset “as it was analysed by the statistician with the potential modifications to ensure full anonymity.”
Although the 6S study was substantially smaller than CHEST, its 800 patients were sicker than those in the Australian study and had a much higher rate of events such as death. Collectively, the two studies have driven the results of systematic reviewers, contributing 80% of the weight to the Cochrane meta-analysis.
Legacy of Joachim Boldt
The inaccessibility of data, lack of interest in raw data among systematic reviewers and regulators, and trust in publications in the story of HES solutions is particularly striking considering that it occurs in a specialty marked by one of the most prolific cases of research fraud under Joachim Boldt (box 2). At the latest count, the website Retraction Watch lists 94 papers, making the German anaesthesiologist the second most retracted author ever.
But perhaps the ultimate irony in the story of the CHEST trial comes from Myburgh himself. Commenting on the fraud of Joachim Boldt, Myburgh stated in an editorial that “journals have a key role in verifying that appropriate ethics review processes have been completed and ensuring that only studies of the highest levels of internal and external scientific validity are published.”11
In his own study, however, John Myburgh has prevented this journal from engaging in the very scrutiny he calls for.
The great colloids versus crystalloids debate
The debate over choice of fluid to treat hypovolaemia has waxed and waned for decades,13 particularly since synthetic colloids such as hydroxyethyl starch came to market in the 1960s and 1970s. Proponents of colloid solutions have argued that despite their higher price, their effects last longer and less fluid is required. Reviews of the evidence have led to highly contrasting conclusions, suggesting at various times that crystalloid solutions14 and colloids15 should be nearing their demise, only to remain in use as the evidence was challenged, and the debate continued. Today, choice of resuscitation fluid varies widely across the world.16
Despite their ubiquity in medicine, the evidence base for fluid resuscitation options has been complicated by a bewildering number of factors, each of which has been argued to affect the results. The first is the product itself. Crystalloids and colloids are both families of products, with each new product created with the belief that its effects would be better than previously available solutions. The second is the diversity of patients, which includes burn victims, trauma patients, those with sepsis, and patients having elective surgery. The safety and efficacy profile of fluids may differ by type of patient.17 18 Other factors have included the lack of high quality randomised trials in conditions that mimic usual care and the dearth of large trials. The difficulty was only compounded when one of the most prolific advocates of colloids—Joachim Boldt—was stripped of his professorship as dozens of his publications were withdrawn from the scientific literature.19
Competing interests: I have read and understood BMJ policy on declaration of interests and declare YODA paid all expenses for me to join a June 2012 planning meeting to discuss YODA’s forthcoming data sharing policy. This payment was later reimbursed by The BMJ. I consider the YODA team to be colleagues. Also, I am an unpaid member of the IMEDS steering committee at the Reagan-Udall Foundation for the FDA, which focuses on drug safety research.
Provenance and peer review: Commissioned; externally peer reviewed.