Comparison of rates of safety issues and reporting of trial outcomes for medical devices approved in the European Union and United States: cohort study

The recent article by Thomas J. Hwang, et al. analyzing rates of safety issues in products approved first in the EU rather than in the U.S. presents interesting data, but it is difficult to draw conclusions about the relative safety of the two approval systems based on their analysis. Moreover, the policy discussion suggests that the rigor of the EU system needs to be improved, while proposals to reduce delays in the U.S. approval system are suspect. Neither of these suggestions is supported by the data presented.

There are a number of questions to be resolved before conclusions can be drawn. First, the authors did not use a classification system for recalls and alerts that would provide comparability between U.S. and EU actions. Second, they fail to identify the root causes of the recalls studied. Third, the higher rates of recalls for products approved first in the EU may simply be an artifact of the types of devices for which the lag in approval times is most significant.

Regarding policy implications, the suggestion that measures to facilitate more rapid review of devices in the U.S. would have a negative impact on safety incorrectly assumes that proposals currently being discussed would lower the standard for review. Moreover, in criticizing the European system, the authors fail to assess the human cost of delayed access to the most important new treatments. The data they present shows that the products most important to patients (“major innovations” in the study’s terminology) were the ones that had the longest lag between availability to patients in the EU and patients in the U.S.

Lack of comparability between recall types

In the U.S., “recalls” fall into three categories. Class I—the most serious—is a situation where there is “a reasonable probability . . . [of] serious adverse health consequences or death.” Class IIs are problems that “may cause temporarily or medically reversible adverse health consequences or where the probability of serious adverse health consequences is remote.” Class IIIs are “not likely to cause adverse health consequences.” In the U.S. a “recall” may involve taking a product off the market, but it may also involve other actions ranging from a warning to physicians to a labeling change to a manufacturing correction. The European system does not distinguish safety notices in the same way. It appears that the authors’ tried to exclude recalls that would be Class III recalls in the U.S., but did not otherwise try to match the severity of the problem provoking the recall in the EU vs. the U.S. Since Class II recalls represent the great bulk of corrective actions in the U.S., the failure to carry out “an apples to apples” comparison is a major problem in interpreting their results.

A study by the Boston Consulting Group for the period 2005-2009 looked only at recalls that were Class I in the U.S. and EU recalls that would have been Class I under the U.S. classification. It found that the number, rapidity, and therapeutic area of recalls was similar in the U.S. and E.U. (1) Thus, without an analysis of the importance of the recalls in protecting human health, it is difficult to assess the significance of the finding that products approved in the EU first were more likely to be recalled.

Lack of a root cause analysis

The BCG study found that more than half (54%) of the Class I EU recalls were for flaws that could not have been discovered in the review process and were classified as “post-market.” A study of U.S. recalls found a similar pattern: Less than half of Class I recalls for both products cleared through the 510(k) path (45.3%) and the PMA path (43.8%) were for pre-market issues. (2) Without such an analysis, it is difficult to know whether the “EU first” higher recall rate results from the EU approval system or other factors. The alternative possibility is amplified by the authors’ finding that two-thirds (67%) of the products for which recalls were identified were approved by both systems, and almost two-thirds of these (63%) were approved first in the EU.

Potential for device type artifact

A final issue is the enormous lag, during this time period, between the approval dates in the EU and U.S. for major innovations and for PMA products. The 44 devices that went through the U.S. PMA process had an average lag of 36.3 months; the 54 devices categorized by the authors as major innovations had an average time lag of 25.1 months (there is presumably some overlap). PMA products, which are the highest-risk products, are significantly more likely to be recalled than 510(k) products. This does not indicate that the PMA approval process, which is extremely rigorous, is flawed: rather, any problem arising with a PMA product is far more likely to present a risk to human health than a problem arising from a product that is less likely to be life-sustaining or life-supporting. Thus, the large time lag between approval of these types of products in the EU and the U.S. may be an important cause for the central finding of the study. The authors include a factor in the regression equation that seeks to control for the FDA approval pathway, but since products approved in the EU but not in the U.S. are not characterized by risk, the control is unlikely to account for all the variance due to this factor.

Policy Implications

The EU is taking steps to improve the consistency of its reviews and to increase, for the highest-risk products, its evidentiary standards for review. The U.S. is working to reduce review times for both 510(k) and PMA products and to encourage first introduction of important new products in the U.S. In taking these steps, it is important that the EU not institute policies that unduly delay the availability of important new treatments to patients in the EU and the U.S. needs to maintain strong review standards that appropriately balance risk and benefit. These goals are eminently achievable.

####

(1) Scott David, Erik Gilberston, and Simon Goodall, “EU Medical Device Approval Safety Assessment: A Comparative Analysis of Medical Device Recalls 2005-2009,” Boston Consulting Group, January 2011. accessed at
https://www.bcgperspectives.com/Images/EU_Medical_Device_Approval_Safety...

(2) Professor Ralph Hall, “Using Recall Data to Assess the 510(k) Process,” paper originally presented to the Institute of Medicine, July 28, 2010, available as testimony before the Senate Aging Committee, April 13, 2011. accessed at http://www.aging.senate.gov/imo/media/doc/4132011.pdf.

It's going to take me a long time to work thoroughly through this article in my "spare" time, so I will not be able to post the results of my full assessment within BMJ's 14-day window for comments here. However...

If I understand the methodology correctly, it seems that the study may have simply confirmed the fact that the more times an event has the opportunity to occur, the more times the event is likely to occur. In this case, the longer a device is on the market, the more times it is used, and the more opportunities it has for a safety event to occur. During the many months that PMA devices were not yet on the US market, but already approved in the EU, they were available for use in the EU. Safety events had the opportunity to occur in the EU, and they did. Conversely, however, during all those months the "approved in the US first" devices were not yet on the US market, they were not on the EU market either. So there was no opportunity for a safety event to occur, and none did.

This also strikes me as potentially problematic:

"The unadjusted rate of safety alerts and recalls in devices approved first in the EU (including CE marked devices not yet approved in the US) was 27% (62 of 232) compared with 14% (11 of 77) for devices approved first in the US (risk ratio 2.1, 95% confidence interval 1.0 to 3.4; P=0.04)."

It doesn't seem appropriate to include CE marked devices not yet approved in the US without also including FDA approved or cleared devices not yet approved in the US. Moreover, this omission would seem to give yet more opportunity for safety events to have occurred for the "EU first" devices. And generally seems to make the study rather incomplete and one-sided, I think?

Amirah, you might consider holding off until the device regulatory community has had a chance to review it.
Please feel free to contact me to discuss further.
Julie at class3devices dot com