Intended for healthcare professionals

Feature Investigation

Rivaroxaban: can we trust the evidence?

BMJ 2016; 352 doi: (Published 03 February 2016) Cite this as: BMJ 2016;352:i575
  1. Deborah Cohen, associate editor, The BMJ
  1. dcohen{at}

An investigation by The BMJ has uncovered the use of a faulty device in a regulatory drug trial, potentially putting patients at unnecessary risk, Deborah Cohen reports

Doctors and scientists are calling for an independent investigation into the key trial underpinning use of rivaroxaban to prevent ischaemic stroke in non-valvular atrial fibrillation after The BMJ found that a defective point of care device was used in the warfarin arm of the trial.

Doctors and scientists have also told The BMJ that the validity of the trial—called ROCKET-AF and published in the New England Journal of Medicine in 20111—is in question until such independent analysis is done.

The drug was manufactured by Bayer and marketed in the United States by Janssen, part of Johnson and Johnson, and the companies relied on a single trial–ROCKET-AF—to gain approval from the US and European regulators. The trial included over 14 000 patients and found that rivaroxaban was non-inferior to warfarin for preventing ischaemic stroke or systemic embolism. There was no significant difference between groups in the risk of major bleeding—although intracranial and fatal bleeding occurred less often in the rivaroxaban group.

But there are now concerns about these outcomes. In a letter submitted to the NEJM (as yet unpublished) and shown to The BMJ, former FDA cardiovascular and renal drug reviewer, Thomas Marcinicak, says: “The care for the warfarin control arm patients [in ROCKET-AF] appears to have been compromised.”

Earlier last year, The BMJ found that the point of care device used to measure international normalised ratio (INR) in patients taking warfarin in ROCKET-AF had been recalled in December 2014. An FDA class I recall notice (the most serious kind) said that certain INR devices could deliver results that were “clinically significantly lower” than a laboratory method. It added that Alere—the device manufacturer—had received 18 924 reports of malfunctions, including 14 serious injuries. The company confirmed to The BMJ that the fault went back to 2002, before the ROCKET-AF trial started.

A falsely low reading could mean that patients had their warfarin dose unnecessarily increased, leading to a greater risk of bleeding. In terms of the trial results, it could make rivaroxaban seem safer than it was in terms of the risk of bleeding and throws doubt on outcomes used to support the use of the world’s best selling new oral anticoagulant.2

Direct oral anticoagulants

Rivaroxaban is a factor Xa inhibitor and belongs to a class of medicines known as the direct oral anticoagulants (DOAC), which also includes dabigatran, apixaban, and edoxaban. They have gained popularity in place of warfarin for the prevention of ischaemic stroke in non-valvular atrial fibrillation because routine blood monitoring is not required.3

Back in September 2015, The BMJ asked the investigators named in the NEJM paper about the recall. They included researchers from Bayer, Johnson and Johnson, and the Duke Clinical Research Institute, which carried out the trial on behalf of the drug companies.

None of the authors responded, but a spokesperson for Johnson and Johnson contacted The BMJ to say that they were “unaware of this recall” and they took the journal’s concerns “seriously.” But it took months of probing by The BMJ before the companies, world drug regulators, and Duke began to investigate the problem in earnest.

Joining the dots

As for the regulators, when The BMJ contacted the European Medicines Agency in April 2015 and subsequently the Food and Drug Administration, both said they did not know that the recalled device had been used in ROCKET-AF. It’s new territory for the regulators. What happens to a pivotal drug trial when a device used is found to be defective?

In November the EMA told The BMJ it was investigating, and the agency subsequently told journalists: “Due to the defect it is now thought that the INR device may have impacted the clotting results in some patients in the warfarin group.”4

Executive director of EMA, Guido Rasi, also called for further independent investigation into direct oral anticoagulants. “It would be nice to have some independent study carried out to give confidence in the use of this medicine,” he said.

The FDA also told The BMJ that it is “aware of concerns regarding the INR device and its use in the ROCKET-AF trial and is reviewing relevant data.” It subsequently announced that it will hold a public workshop about the safety and effectiveness” of point of care INR devices in March “to seek and identify potential solutions” to what it said were “scientific and regulatory challenges.”

However, in the meantime spokespeople for Johnson and Johnson and Bayer issued identical statements in December 2015: “We have conducted a number of sensitivity analyses. These sensitivity analyses confirm the results of the ROCKET-AF study and the positive benefit-risk profile of Xarelto (rivaroxaban) in patients with non valvular atrial fibrillation.”

But what should happen amid the uncertainty?

Harlan Krumholz, professor of medicine (cardiology) at Yale University, says that the NEJM should place an “immediate expression of concern” on the paper to notify the medical community.

“The study should be considered of uncertain validity until a more thorough review can be done,” he says, adding that there should be “an investigation by an independent group of experts to quickly determine if there are grounds for retraction.”

Concerns about warfarin control

Even before rivaroxaban was approved in Europe and the US in 2011 for use in non-valvular atrial fibrillation, regulatory officials raised concerns about the warfarin control in the ROCKET-AF trial. Two primary clinical FDA reviewers of the drug recommended that it should not be approved for the US market.

“ROCKET provides inadequate information to assess the relative safety and efficacy of Xarelto in patients whose warfarin administration can be well-controlled,” they wrote in an FDA decisional memo—which outlines clinical reviewers’ view on whether a drug should be approved.5

However, they were seemingly unaware that there are other reasons to be concerned about the adequacy of the warfarin control in the ROCKET-AF trial that have since emerged.

Lack of transparency over devices in trials

Currently, there is little public information about which diagnostic point of care devices are used in any of the direct oral anticoagulant trials (box). They are not named in the published phase III trials. The BMJ became aware that the problematic device was used in the ROCKET-AF trial only by reviewing European regulatory documents in April last year.

Devices used in other trials

Given the lack of publicly available information about the point of care testing devices used in the other direct oral anticoagulant trials, The BMJ sought to find out what they are.

Lars Wallentin, corresponding author of the phase III ARISTOTLE trial (Apixaban versus Warfarin in Patients with Atrial Fibrillation)6 said that the trials used the ProTime POC device made by International Technidyne Corporation, Edison, NJ, USA.

Daiichi-Sankyo, the manufacturers of edoxaban, also said that the ProTime POC device was supplied to all study sites in the Edoxaban versus Warfarin in Patients with Atrial Fibrillation Trial (ENGAGE AF)7 and in its venous thromboembolism trial.

Marciniak says that the NEJM, which published the trials for three of the direct oral anticoagulants, should rectify that.

“You should require that the devices used in trials are clearly and specifically identified in your publications,” he wrote in his letter.

How has this come to happen?

In tracking the faulty recall and its potential effect on the outcomes of a global clinical trial, The BMJ has once again come across flaws in device regulation. A series of journal investigations have highlighted the lack of clinical data required by US and Europe regulators for high risk implants, such as metal on metal hips, before they are put on the market.8 They have also shown how slow regulators can be to act when problems do emerge and shown how oversight can be lacking on the performance diagnostic tests.9 10

In 2005, a warning letter from the FDA to HemoSense—the company that marketed the faulty device before Alere bought it—reprimanded them for failing to investigate “clinically significant erroneous” high and low INR results generated by the point of care device.

“Both high and low test [INR] results have the potential to cause or contribute to a death or serious injury, because: they may result in erroneous dosing and thus improper control of coagulation,” the letter said.11

Despite these warning letters, the FDA cleared subsequent iterations of the device through its 510(k) regulatory system. This system requires makers of such devices to show only that the new version is “substantially equivalent,” or similar, to one already on the market. It has been criticised by the likes of the Institute of Medicine for not providing enough evidence that a device is safe and effective.12

Johnson and Johnson, however, has lobbied against tightening up this aspect of device regulation and the need to provide more evidence.13 But the lack of a regulatory requirement for the diagnostic accuracy of the device to be checked before it came on to the market has allowed the fault to creep through the system.

Alere has confirmed to The BMJ that the fault dates back to 2002 and it may occur in all devices and not just one batch. However, neither it nor the FDA responded to questions about why nothing had been done about the problem earlier.

Were the companies aware of any problems during the trial?

The BMJ asked Johnson and Johnson, Bayer, and Duke if any investigator complained to them about mismatched point of care and laboratory INR readings if someone had a bleed in the trial. The BMJ also asked if they had validated the device at any point before or during the trial. None responded to the questions.

According to former FDA clinical pharmacologist, Bob Powell, who has also worked with industry and academia, the specificity and reproducibility of a diagnostic test or assay is vital to the performance of a trial.

“The fact that this was apparently not previously done nor reported in the primary publication is concerning as this is a basic principle in drug development,” he says.

What next?

The EMA has told The BMJ that it has asked the companies for analyses and would consider any analyses by Duke too. During the trial INR at 12 and 24 weeks was measured at a central laboratory as well as with the point of care device. Powell says that “a comparison should be made between the defective point of care readings and the two sets of ‘gold standard’ central lab readings” as this would “determine whether this defective device undermined the integrity of the trial results.”

It is not clear that this has happened. In December last year, Duke issued a press release with a summary report of the results of their “secondary analysis of the trial findings.”

“The findings from the analysis are consistent with the results from the original trial and do not alter the conclusions of ROCKET-AF—rivaroxaban is a reasonable alternative to warfarin and is non-inferior for the prevention of stroke and systemic embolism with less intracranial hemorrhage and fatal bleeding” it said.

But Powell says this statement is “misleading” because of the lack of information.

Krumholz also thinks that this statement did not give enough information about what Duke found in terms of the major safety endpoint—major bleeds.

“The DCRI is among the most respected research institutions, but this statement suggests that they know important information that relates to the ROCKET-AF trial but are delaying in disseminating the information until it can be published,” he says.

Hugo ten Cate, medical director of the Maastricht thrombosis anticoagulation clinic and coeditor in chief of Thrombosis Journal, says that major bleeds have serious consequences.

“Large bleeds mostly occur in the gastrointestinal tract and can be lethal if substantial blood loss occurs, especially in elderly subjects with comorbidity; this can be a devastating complication,” he says.

Any changes to the ROCKET-AF trial will have a broader effect on the literature.

Carl Heneghan is an author on a forthcoming Cochrane Collaboration review of “direct thrombin inhibitors and factor Xa inhibitors for atrial fibrillation,” which includes the ROCKET trial.

He has written to Duke to ask if the results for the main outcome measures in the reanalysis are the same as in the original published paper and, if not, what the differences are after the reanalysis.

A spokesperson for Duke did not answer the question but said that the ROCKET-AF executive committee “intends to publish a full description of its analysis as rapidly as possible.”

Independent oversight

But given the lack of clarity over the outcomes and the methods used, is a reanalysis by Duke enough?

Marciniak is unequivocal. He says that he would not rely on any reanalyses done by Duke, Johnson and Johnson, or the FDA.

“Because they already missed the problems both in the trial and with the public marketing, I would not trust them to publish anything that is accurate—or that provides any details,” he told The BMJ.

He added that the datasets need to be released as “the only solution that would lead to unbiased analyses.”

But previous attempts to do this have been thwarted.

Krumholz has approached Johnson and Johnson for access to the trial data. His Yale University Open Data Access (YODA) project has an agreement with Johnson and Johnson to make all of the clinical trial data available for its approved products. However, although the company agreed to allow access to the data, Bayer refused.

“This is an ideal situation for data sharing. The evaluation of the data in this trial should not go on behind the curtain. And it seems imprudent to allow those who conducted the trial to be the only ones who can touch the data,” Krumholz says.

But it doesn’t look like the data release is going to be sanctioned by Bayer any time soon. A spokesperson for the company told The BMJ that this is because they have signed up to sharing information only on “study reports for new medicines approved in the US and the EU after January 1, 2014.”

The request does not fit in their “current scope of clinical trial data sharing.”

Good outcome for patients?

But in the end might this series of errors lead to a favourable outcome for the regulators—and perhaps patients?

At the end of 2015, both the EMA and the FDA held meetings to discuss the need to measure blood levels of direct oral anticoagulants and adjust the dose accordingly to maximise benefit and minimise harm—despite all the manufacturers claiming that this is not necessary. The meetings were held after The BMJ revealed that Boerhinger Ingelheim, manufacturers of dabigatran, withheld analyses from the regulators that showed how many major bleeds could be prevented by monitoring anticoagulant activity and adjusting the dose.14

A presentation to EMA last year by Robert Temple, deputy director for clinical science at the FDA’s Center for Drug Evaluation and Research, suggests that the FDA believes there is a scientific argument for measuring the blood levels of these drugs and adjusting the dose.

“Being too low leads to a stroke, a very bad outcome, and being too high leads to major bleeds, also bad, so that early optimization [of the dose] seems worthwhile,” he said adding that direct oral anticoagulants are “very good, but could probably be better.”

But once a drug is on the market, regulators lack a mandate to act unless there are safety concerns. However, according to Powell, depending on the outcomes of any reanalysis of the ROCKET-AF trial, this might allow them to take action.

“After a drug is approved, it usually takes a safety signal to prompt significant action on the part of the FDA. It is this lack of safety signal that appears to be hindering the FDA in their desire to pursue tailored dosing for DOACs. If it turns out that the issue with the [INR] device changes the safety profile of rivaroxaban, this may constitute the safety signal necessary for the FDA to act in this regard,” he said.


Cite this as: BMJ 2016;352:i575


  • Competing interests: I have read and understood BMJ policy on declaration of interests and have no relevant interests to declare.

  • Provenance and peer review: Commissioned; externally peer reviewed.


View Abstract