Feature Investigation

Manufacturer failed to disclose faulty device in rivaroxaban trial

BMJ 2016; 354 doi: https://doi.org/10.1136/bmj.i5131 (Published 28 September 2016) Cite this as: BMJ 2016;354:i5131
  1. Deborah Cohen, associate editor, The BMJ, London, UK
  1. dcohen{at}bmj.com

An investigation by The BMJ finds that companies were aware of concerns about a faulty device in a regulatory trial and reveals data that suggests participants were put at unnecessary risk. Deborah Cohen reports

A drug manufacturer knew about problems with a blood testing device but did not share data before the crucial approval process, an investigation by The BMJ has found.

Janssen, the pharmaceutical arm of Johnson and Johnson, withheld data from the Food and Drug Administration about problems with the INRatio device, which was used in the phase III trial (ROCKET AF) of the blockbuster anticoagulant.

The company generated these data in a safety programme (called the Covance recheck programme) set up after trial investigators became concerned, shortly after the study began, about the accuracy and reliability of the point-of-care device used to monitor patients receiving warfarin.

Janssen also failed to share these data with the safety monitoring board of the trial.

Executives from Bayer—which codeveloped rivaroxaban and markets it outside the United States—were also aware about concerns about the device used in the trial. However, the German company did not know about the existence of the recheck programme until this year.

Patients in the US are suing Janssen and Bayer for allegedly misleading users about the safety and efficacy of rivaroxaban (Xarelto).

In legal testimony, a Bayer official has alleged that Janssen, which had responsibility for conducting and managing the trial, withheld the programme from the company.

When asked by The BMJ, Bayer stated that it “Expressly contradicts with the allegation that Bayer would have withheld safety data from drug regulators and the safety monitoring board of the ROCKET AF trial.”

Pivotal trial comes under scrutiny

Published in the New England Journal of Medicine (NEJM) in 2011, the trial included over 14 000 patients and concluded that rivaroxaban was “similar to warfarin in its ability to prevent” ischaemic stroke or systemic embolism in people with non-valvular atrial fibrillation.1

The authors reported there was no significant difference between groups in major bleeding risk—although intracranial and fatal bleeding occurred less often in the rivaroxaban group.

The point-of-care device INRatio—initially marketed by HemoSense and later by Alere—was used to measure international normalised ratio (INR) values in the 7133 participants in the warfarin arm of the study. However, the FDA recalled the device in December 2014 because INR results it generated could be “clinically significantly lower” than those found by a laboratory method.2

However, Janssen and Bayer did not tell the authorities that the device used in the ROCKET trial had subsequently been recalled until The BMJ probed them in September 2015, the New York Times reported.3

Global regulators have subsequently launched inquiries; Bayer and Janssen have reanalysed the trial data; the study’s executive committee has published two letters in NEJM containing its “independent” reanalyses of the data4 5; and the US Department of Justice has issued Alere with a subpoena seeking “various documents related to the accuracy, reliability and performance of the INRatio system.”6

Only the European Medicines Agency has published its full conclusion stating, “Any incorrect measurements obtained with the defective device would have had only a marginal effect on the study results, and the safety of Xarelto remains unchanged.”7

A spokesperson for the FDA told The BMJ that it is “continuing to review relevant data,” but said that it has “not changed its recommendations regarding the use of Xarelto.”

The FDA also said that because the INRatio devices were used only to monitor blood clotting rates and adjust the dose of warfarin in the trial, they are “confident that the rates of stroke, bleeding, and other clinical outcomes” in patients taking rivaroxaban are correct.

However, the relative risk of stroke and bleeding of rivaroxaban compared with warfarin “could be affected” by the performance of these devices.

Could the regulatory investigations have been avoided?

Markku Kaste, one of the trial investigators and former head of the Clinical Stroke Research Group at Helsinki University Central Hospital, was worried about the use of the INRatio device and one of the earliest to ask questions.

Kaste told The BMJ that he warned a study monitor from Parexel, which had responsibility for overseeing investigational sites outside North America, that INRatio’s performance should first be validated. According to Kaste, the monitor reassured him that there was “nothing to worry about.”

The BMJ asked Parexel what it knew about the malfunctioning device and the Covance recheck programme.

A spokesperson said: “I cannot discuss the specifics of a study due to Parexel’s confidentiality obligations to our clients.”

Yet several physicians raised concerns that the devices were providing inaccurate results.

Shortly after the trial started in February 2007, members of the trial’s executive committee emailed Janssen questioning the accuracy and reliability of the INRatio. Following complaints from other trial investigators about the device, Janssen launched the Covance recheck programme in early 2008.

In a letter from Janssen to investigators dated 21 February 2008, the company described a new kit for collecting “special blinded INRs … designed to assist investigators who believe that a subject’s INR values obtained with HemoSense point of care (POC) device [INRatio] are greatly different from what was expected.”

Janssen’s letter did not describe the concerns that led the company to send the special kits. Nor was the new recheck programme ever added to the trial protocol (although EMA disputed this point, after confirming that the first it heard of the recheck programme was when The BMJ notified the agency in May 2016). Bayer has told The BMJ that: “This support program was in addition to the use of an unblinded INR monitor which was described in the protocol for the trial.”

Yet in March 2016, Janssen’s lawyers described the recheck programme as a “component” of the ROCKET trial.

Both Bayer and Janssen have told The BMJ that investigators submitted 149 samples to the recheck programme (rivaroxaban 78; warfarin 71). In Janssen’s legal pleading, it stated that there were “16 instances where the value from the point of care device and lab were recorded as inconsistent.”

Since the rivaroxaban arm used a sham device, this means that 23% of warfarin participants in the programme had mismatching laboratory and point-of-care INR values.

However, it is unclear how Janssen defined “inconsistent” readings, and the company did not explain when asked by The BMJ.

Perhaps most alarming, Janssen did not tell the FDA about the recheck programme or trial investigators’ concerns about the device during the approval process in 2011.

Who was informed?

Despite the safety of trial participants potentially being compromised, The BMJ’s investigation shows that Janssen did not give the data generated by the recheck programme to the trial’s data and safety monitoring board.

Peter Rothwell, professor of clinical neurology at Oxford University, was a member of the board. He told The BMJ that he has “no memory of the board being told about the programme.

“Clearly, if the sponsor of the trial had concerns about the validity of the point-of-care testing of INR that would have been important for the board to be made aware of [them],” Rothwell said.

Kaste told The BMJ that the data from the Covance recheck programme should have also been shared with investigators as “it’s vital for the ongoing ethical approval for the trial.”

“In my institution I have to renew ethical approval annually and it is questionable if they would have given their approval if they knew about the problems with the device,” he said, adding: “I now have some doubts about the validity of ROCKET AF trial. It possibly skews the data in favour of rivaroxaban.”

Hiding in plain sight

There may be reason to believe regulators could have caught the problem themselves.

As part of the ROCKET trial design—which is online at NEJM.org—a split sample was taken from participants at weeks 12 and 24 of the trial. One sample was analysed by the point-of-care device and the other by a central laboratory. The samples were taken for pharmacodynamic studies and the results were kept blinded until the trial was completed.

The 12 and 24 week paired sample data were not used to ascertain the accuracy of the device before the drug was approved, but since questions have been raised, they are central to the current debate over the device’s accuracy and its implications for the validity and generalisability of the trial.

Reviews submitted to the EMA by Bayer and Janssen “each determined that the potential issue with Alere’s monitoring device did not impact the conclusions of the ROCKET-AF trial,” a spokesperson from Janssen told The BMJ in a statement adding: “The benefit-risk profile of Xarelto remains positive and unchanged for reducing the risk of stroke in patients with non-valvular atrial fibrillation.”

However, these analyses did not include all the patients. When the device was recalled in December 2014, the recall notice listed specific patient populations that were purportedly more likely to have faulty readings. The companies focused their analyses on these subgroups.

Thomas Marciniak, a former FDA official who was a drug reviewer on Janssen’s application to use rivaroxaban in acute coronary syndrome, told The BMJ that these analyses are “worthless” because the “inaccuracies are not limited to the recall patients.”

And EMA’s review—published in February 2016—seems to corroborate Marciniak’s concern. Its analysis of the week 12 and 24 paired samples found “discrepancies of potential clinical relevance” in about 35% of the estimations.

The target INR range during the trial for people taking warfarin was 2-3. However, EMA’s report shows at week 12, for more than 28% of laboratory INR readings above 4 the corresponding INRatio readings were below 3 three (and some were below 2). Documents obtained under freedom of information by The BMJ show that this rose to 33% at week 24.

EMA’s report further states that warfarin may have been improperly dosed in participants whose INRatio readings were lower than the corresponding laboratory value.

It’s a point with which the FDA concurs. A spokesperson told The BMJ: “The information that has been developed to date indicates that some patients may have received higher warfarin doses than needed because of the use of INRatio devices in this trial.”

To estimate the effect of the misreading, EMA asked Bayer to compare event rates between those participants whose laboratory results fell within range of the INRatio reading and those whose results were out of range, by different degrees. These analyses found that the larger the difference between the INRatio and laboratory readings, the higher the rate of major bleeding. People whose readings were more than 2 units apart were over 40% more likely to have a major bleed than those whose readings were the same.

Carl Heneghan, professor of evidence based medicine at Oxford University and an author on a forthcoming Cochrane review of direct thrombin inhibitors and factor Xa inhibitors for atrial fibrillation, believes patients in the trial may have been put at undue risk of harm.

He told The BMJ that the INR device errors “are worrying” as there is “a near exponential increase in bleeding risk with increasing INR.

“With INRs above 4, dose adjustment is required. INRs above 5 require investigation and doses to be withheld, and because of the defective readings this will not have happened. More worrying is some of the normal results were actually INRs above 8, which require active intervention to reduce the risk of bleeding.”

Despite all this, the EMA report stated “that there was sufficient evidence to conclude that the benefit/risk balance remains unchanged and favourable for treatment with rivaroxaban in the prevention of thromboembolism in non-valvular atrial fibrillation.”7

Marciniak described EMA’s review as a “whitewash,” alleging that the regulator has ignored “the serious device inaccuracies that those analyses reveal.”

Former FDA clinical pharmacologist, Bob Powell, who has also worked with industry and academia, told The BMJ that he is “disappointed” with EMA’s conclusion. “The decision not to change the label [of rivaroxaban] does not accurately describe the uncertainty around the trial results.”

The FDA says it is still investigating.

FDA’s concerns before approval

Marciniak says that the FDA should have identified the problems with the 12 and 24 week data. The division of FDA responsible for reviewing the ROCKET AF data had concerns about the performance of INRatio.

In March 2011, while the drug was under review, the FDA asked Janssen for information about the accuracy of the device. But according to plaintiffs’ legal filings, the company did not provide the evidence from the Covance recheck programme nor the thousands of paired laboratory samples.

Janssen did not respond to The BMJ’s questions about this. The company instead sent a statement saying: “We have acted with urgency, diligence and in the best interests of patients and prescribers, sharing data with health authorities and the safety monitoring board of the ROCKET-AF trial.”

NEJM reanalysis

Bayer and Janssen have highlighted the “independent reanalysis” performed by the ROCKET AF executive committee in conjunction with the Duke Clinical Research Institute to support their conclusions that the device malfunction did not affect trial outcomes. This was published as a letter in the NEJM in February 2016.4

But this letter did not include an analysis of the paired week 12 and 24 data—despite a peer reviewer of the letter asking, “What percentage of major bleeds in the warfarin arm had an elevated INR as measured by a lab-based plasma INR?”

Manesh Patel, associate professor of medicine at Duke Clinical Research Institute and lead author of the published trial and letter, told the NEJM editors that they “are unable to do this analysis as we do not have systematic measurement of both point-of-care and laboratory values throughout the course of the study.”

However, in court filings from earlier this year, Janssen’s lawyers seemed to contradict this position, stating that the paired laboratory and INRatio data were available to the authors.

Neither Patel nor Fox responded to The BMJ’s questions about this.

After a letter of complaint by Powell to the NEJM in July,8 stressing the laboratory data at 12 and 24 weeks should be compared with the point-of-care data from the split samples, Patel and colleagues reanalysed the data again. They calculated that 13% of warfarin patients had discordant results at 12 or 24 weeks and 4% at both, and concluded that the new results were “consistent” with their original trial report.5

But much of the debate hinges on what degree of inaccuracy is considered acceptable. In the case of INR point-of-care devices, the FDA has required at least 90% agreement with laboratory INR results, but there are signs the agency may soon tighten its rules (box).

Current point-of-care device standards not stringent enough, says FDA

Current international standards set a level of acceptable mismatch between the point-of-care device and laboratory results.

According to 2007 ISO guidance, a point-of-care device should report INRs within 0.5 of laboratory value (for INRs <2) or within 30% (for INRs between 2 and 4.5), at least 90% of the time.

Following a “high number” of medical device reports of malfunction, serious adverse events, and patient deaths, the FDA has said it will review current standards.

At a meeting in March, Rong Rong, medical officer in the haematology branch in the FDA’s Division of Immunology and Hematology Devices, said that accurate INR measurement was critical to the safety and effectiveness of warfarin. Over the past two years, the FDA received reports of 3231 malfunctions, 384 serious injuries, and 18 deaths for INR point-of-care devices. Rong is concerned: “Inaccuracy in INR measurement can have serious consequences and can cause patient death.”

Point-of-care INR devices are currently classified as a moderate risk class II device cleared through the FDA’s 510(k) process. This means that they do not require clinical studies to prove their safety and effectiveness.

Rachel Goehe, also from the FDA, told the March meeting that some manufacturers want the FDA to continue to recognise the ISO standard. However, she said: “The FDA does not recognise the standard for need of professional use nor patient self-testing, because the allowable bias limits are too large and may impact management of patient care.”

She added the agency had proposed to tighten the standard to reduce the allowable bias and add recommendations when to monitor the devices performance against laboratory tests.

Goehe stressed the FDA’s need “to obtain postmarket surveillance data for adverse events while using a wide variety of sources and methods” in case of device malfunction.

However, Jay Ronquillo a physician and engineer speaking on behalf of the National Center for Health Research, argued that the ROCKET AF trial “has raised questions about the comparative safety of Xarelto. This shows how device malfunctions no longer affect just single users but directly influence approval decisions for any new drug being compared to warfarin.”

He urged the FDA to consider putting point of care PT/INR devices through the agency’s more rigorous premarket approval process.

“[These devices] represent an increasingly important part of the healthcare landscape. Transparent and robust design, performance studies and clear data supporting safety and effectiveness are required to avoid device problems that could have wide reaching consequences for patients and public health,” he said.

Patel and his coauthors also reported that in warfarin treated participants with discrepant INRatio and laboratory results, both ischaemic stroke and bleeding rates were higher than in those with non-discrepant results. They argued that the rise in both types of events—not just bleeding—countered the hypothesis that device malfunction would have led to increased clinical events.

However, Powell told The BMJ that some of the ROCKET team’s assumptions were wrong: “The incidence of both ischaemic stroke and bleeding increase with warfarin treatment as the INR goes above 4.”

He added that the discrepancies in the INR readings from INRatio did not occur in the same patients at 12 and 24 weeks, and this variation would have occurred throughout the study.

Some are also questioning how “independent” these analyses actually are, particularly since employees of both Janssen and Bayer are members of the executive committee that did the reanalysis.

Both Janssen and Bayer may have known the extent to which INRatio malfunctioned before the letter was submitted for publication. US plaintiffs’ lawyers allege that on 21 September 2015, Janssen and Bayer held an internal meeting that showed 32% of the paired laboratory and point-of-care device samples obtained by the INRatio were off by 0.5 units or more.

A spokesperson for Bayer said it and Janssen “did not attempt to influence the ROCKET AF executive committee re-analysis data submission to the NEJM.” However, in correspondence with the editors of the NEJM, Patel said that “the entire executive committee and team on the letter helped revise and work on the data response.”

Patel and Keith Fox, professor of cardiology at Edinburgh University and chair of ROCKET AF, provided a statement on behalf of the trial’s executive steering committee.

“The ROCKET AF Executive Committee (EC) stands behind the conduct and rigor of the study, the trial findings, and the recently published research letter,” they said.

Can we trust ROCKET at all?

The problems with INRatio are not the first ROCKET has faced, and people are now questioning if regulators can salvage the trial. During the drug’s assessment, two FDA clinical reviewers recommended rivaroxaban not be approved because of inadequate warfarin control in the trial. The mean time in therapeutic range was 55%—lower than in any of the phase III trials of other direct oral anticoagulants.

Steven Nissen, a cardiologist at the Cleveland Clinic who served on the FDA advisory panel and voted against approving rivaroxaban in 2011, told the New York Times that: “Given the fact that the device was inaccurate, there is no way anybody can tell you what would have happened in the trial.”

While the FDA continues its investigation, which will include a surveillance study using electronic health records to compare rivaroxaban with warfarin, it told The BMJ that the device malfunction didn’t have a “significant impact”—but it’s as yet unclear just what this means.

Heneghan remains unsure. “It is impossible to create subgroups that had ‘accurate results' based on just two external quality control measures. In addition, the measures of TTR [time in therapeutic range] quoted in the trial results would be worse if the results were based on a lab INR results,” he said.

“The implications are we still do not know whether this is a safe drug. We need a trial to assess the safety and efficacy of rivaroxaban. Just one caveat—it should be run independently.”

“What The BMJ reported in February 20169

  • INRatio, the point-of-care device used to monitor warfarin in the control arm of the key pivotal trial underpinning rivaroxaban’s approval to prevent ischaemic stroke in non-valvular atrial fibrillation, was faulty and later subject to a class 1 FDA recall

  • The BMJ alerted regulators and the authors of the trial that was published in the NEJM. Janssen said it was the first it had heard of the device recall

  • Janssen and Bayer told The BMJ their own analyses show the device had not affected trial outcomes; EMA and FDA said they were still investigating

  • Doctors and academics called for the data to be released and independently evaluated

What this story adds

  • Janssen executives were aware, early on in the trial, of concerns about device malfunction

  • Bayer executives also knew of concerns about the device in the trial

  • Janssen, which had primary responsibility for the trial, launched an unpublicised safety programme. This involved an “unblinded monitor” who would check INRatio readings against laboratory (Covance) results if an investigator had concerns

  • Janssen continued to use INRatio throughout the trial and did not disclose the presence of, nor the data from, the Covance recheck programme to the trial’s data and safety monitoring board, Bayer, or the FDA before drug approval

  • EMA only learnt about the recheck programme after The BMJ alerted it to it in May 2016

  • Patients in the US are suing both Janssen and Bayer for allegedly misleading users about the safety and efficacy of rivaroxaban. The first trial is scheduled to start next year


2002 FDA 510(k) clearance of HemoSense INRatio point-of-care INR device

October 2004 Paper published showing that of 8 people with hypertherapeutic INR values were misclassified as normotherpeutic by INRatio10

October 2005 FDA sends warning letter to HemoSense stating: “Our review indicates that your firm had information indicating that INRatio devices were generating clinically significant erroneous values”

  • September 2006 Johnson & Johnson submitted the final protocol, case report forms, and other study documents for regulatory approval of rivaroxaban to the FDA

  • November 2006 FDA sends warning letter to HemoSense stating: “These violations include, but are not limited to, the following: Failure to investigate complaints involving the possible failure of a device … For example: … The complaint involved discrepant results between the INRatio INR device and the Lab INR.” (In Janssen’s possession as of January 2007)

  • December 2006 First patient enrolled in ROCKET AF

  • February 2007 Numerous people, including members of the ROCKET AF executive committee, are said to have expressed concern over the accuracy and reliability of the INRatio device and called on Janssen to validate the device or implement study-wide routine quality control procedures, plaintiffs’ lawyers allege

  • July 2007 Bluestein et al publish paper published with the conclusion: “POC devices may not be appropriate for commercial laboratory test substitutions without prior performance evaluation”11

  • February 2008 ROCKET AF investigators told in letter from Janssen to contact the medical monitor, Parexel or Duke Clinical Research Institute helpline if they have concerns about the INRatio device

  • September 2010 Last patient contact in ROCKET AF

  • October 2010 Database lock of ROCKET AF

  • 14 March 2011 FDA issues Janssen information request relating to performance of INRatio device in ROCKET AF

  • 17 March 2011 Janssen’s response to information request submitted, which it is claimed does not include Covance recheck programme data nor week 12/24 paired data

  • 8 September 2011 ROCKET AF published in NEJM

  • 8 September 2011 FDA advisory committee meeting to discuss approving rivaroxaban for non-valvular atrial fibrillation. Clinical reviewers express concern about warfarin’s time in therapeutic range

  • 22 September 2011 EMA grants marketing authorisation for rivaroxaban for the prevention of stroke and systemic embolism in patients with non-valvular atrial fibrillation

  • November 2011 FDA grants market authorisation for rivaroxaban to Janssen for the prevention of stroke and systemic embolism in patients with non-valvular atrial fibrillation.

  • December 2014 FDA initiates class I recall of the INRatio and INRatio 2 stating: “Device may provide INR results that are clinically significantly lower than results obtained using a reference INR system (laboratory method)”

  • February 2015The BMJ contacts the EMA about the recalled device used in ROCKET AF

  • March 2015 Alere confirms to The BMJ that the devices affected by the recall include those used in the ROCKET AF trial

  • September 2015The BMJ asks investigators named in the NEJM paper about the recall

  • September 2015 According to the plaintiffs’ legal documents ROCKET AF Task Force held a meeting where data were discussed indicating that INRatio results were off by 0.5 INR units or more 32% of the time as compared with central lab at two points during the trial (week 12 and 24)

  • 24 September 2015 Alere confirmed to Janssen/Bayer that recall applied to INRatio devices used in ROCKET AF

  • December 2015 Letter containing the “independent” reanalysis of ROCKET AF submitted to the NEJM by the executive steering committee

  • December 2015 Janssen provides the FDA a comparison of the week 12 and 24 paired samples, plaintiffs’ lawyers state

  • 11 January 2016 Duke Clinical Research Institute omits paired INRs from response to NEJM despite being specifically asked by peer reviewers

  • 3 February 2016 Patel and colleagues reanalysis published in NEJM4

  • 3 February 2016The BMJ  publishes “Rivaroxaban: Can we trust the evidence?”9

  • 5 February 2016 EMA assessment report states that: “any incorrect measurements obtained with the defective device would have had only a marginal effect on the study results, and the safety of Xarelto remains unchanged”7

  • 7 March 2016 Janssen provides FDA “documents, which consist primarily of email chains, that demonstrate on numerous occasions physicians involved in ROCKET AF trial raised concerns that the devices were providing inaccurate results,” plaintiffs’ lawyers allege

  • 8 March 2016 Janssen lawyers acknowledge the existence of the Covance recheck programme in the US courts

  • April 2016 FDA confirms it had previously been unaware of the recheck programme. It asks Janssen for a detailed description of its purpose, operational details, data generated by the programme, and information on who knew what and when they knew it.

  • May 2016The BMJ contacts the EMA to ask about the recheck programme. The agency later confirms it’s the first it had heard about it



View Abstract

Log in

Log in through your institution


* For online subscription