Comparison of rates of safety issues and reporting of trial outcomes for medical devices approved in the European Union and United States: cohort studyBMJ 2016; 353 doi: https://doi.org/10.1136/bmj.i3323 (Published 28 June 2016) Cite this as: BMJ 2016;353:i3323
- Thomas J Hwang, researcher1 3,
- Elisaveta Sokolov, neurologist2,
- Jessica M Franklin, assistant professor of medicine3,
- Aaron S Kesselheim, associate professor of medicine3
- 1Faculty of Arts and Sciences, Harvard University, Cambridge, MA, USA
- 2Department of Neurology and Neurophysiology, King’s College London, London, UK
- 3Program on Regulation, Therapeutics, and Law (PORTAL), Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, Boston, MA, USA
- Correspondence to: ,
- Accepted 1 June 2016
Objective To evaluate safety alerts and recalls, publication of key trial outcomes, and subsequent US approval of high profile medical devices introduced in the European Union.
Design Cohort study.
Setting Novel cardiovascular, orthopedic, and neurologic devices approved in the EU through Conformité Européenne marking between 2005 and 2010.
Data sources Public and commercial databases searched up to January 2016 for press releases and announcements of approvals; public Food and Drug Administration and European regulatory authority databases for US approvals and safety alerts and recalls; and Medline, Embase, and Web of Science for peer reviewed publications.
Main outcome measures We categorized the novelty of the devices in the study sample as a “major innovation” or an “other change,” and extracted descriptive data about the devices and information on any safety alerts and withdrawals. Linear regression models examined factors associated with differential EU and US approvals. Cox proportional hazards regression models were used to evaluate factors associated with safety alerts and recalls and the publication of trial outcomes for devices categorized as major innovations. Models controlled for time, therapeutic category, regulatory pathway, size of sponsoring company, and indicator variables for devices approved first in the EU and devices approved only in the EU.
Results 67% (206/309) of devices identified were approved in both the US and the EU, of which 63% (129/206) were approved first in the EU. The unadjusted rate of safety alerts and recalls for devices approved first in the EU was 27% (62/232) compared with 14% (11/77) for devices approved first in the US. The adjusted hazard ratio for safety alerts and recalls was 2.9 (95% confidence interval 1.4 to 6.2) for devices approved first in the EU. The results of pivotal trials were published for 49% (37/75) of devices categorized as major innovations, with an overall publication rate of 37% five years after approval.
Conclusions Devices approved first in the EU are associated with an increased risk of post-marketing safety alerts and recalls. Poor trial publication rates mean that patients and clinicians need greater regulatory transparency to make informed decisions about treatment.
Medical devices play an important role in patient care, but the approval of the devices and their regulation are handled differently in the European Union and United States. In the EU, devices can be marketed if they perform “as intended” and are likely to be safe; clinical testing may be required for some high risk devices. By contrast, in the US, high risk devices must demonstrate reasonable assurance of safety and effectiveness before they can be used by patients, generally through the conduct of prospective clinical trials (see table 1⇓).1
The public health consequences of these diverging paradigms for device regulation have come under increasing scrutiny.2 3 4 5 On the one hand, approving devices without first requiring evidence of safety and effectiveness can reduce the time to market of new technologies. For example, transcatheter aortic valve replacement was developed as a less invasive treatment option for patients with symptomatic aortic stenosis.6 The first transcatheter aortic valve replacement was approved for use in the EU in 2007, four years earlier than the first approval by the US Food and Drug Administration in 2011 (although it was not until 2010 that evidence of this device’s effectiveness was established by a published randomized controlled trial).7
On the other hand, widespread use of new devices without adequate testing makes informed treatment decisions difficult and can slow the ability of investigators to generate this information for future patients. In addition, patients might be exposed to an increased risk of harms from devices of uncertain utility. For example, a report by the FDA highlighted 12 devices that were approved in the EU (but not the US) and later found to be unsafe or ineffective.8 The recent emergence of safety issues involving implanted devices in the EU9 10 11 12 and the US13 14 15 has renewed calls to revisit the appropriate trade-offs between access to devices and risk. After the European Commission called for immediate action to better guarantee the safety of medical technology in the Joint Action Plan of 2012, new draft regulation is now in the late stages of negotiation.
Evidence on the safety and performance of the EU’s approach to the regulation of devices is limited.16 Previous studies have focused on the quality and reporting of evidence supporting approvals of devices in the US3 17 18 19 20 21 22 23 24 and safety record of devices approved with less clinical testing.25 26 27 Attempts to study EU device regulation have been hampered by the lack of a publicly available register of approved devices in the EU (as there is in the US); the confidentiality of information submitted to notified bodies and regulators; and the decentralized nature of approval decisions made across individual member states.1 16 28
We identified high profile new medical devices approved in the EU between 2005 and 2010 and evaluated safety issues related to these devices and publication of the key outcomes of trials. We focused on cardiovascular, neurologic, and orthopedic devices, which account for the majority of high risk devices used in clinical practice.
We used public and commercial sources to construct a novel dataset of medical devices granted marketing approval (Conformité Européenne (CE) marking) in the EU between 1 January 2005 and 31 December 2010 and indicated for the treatment of cardiovascular, neurologic, and orthopedic conditions (fig 1⇓). We focused on “high profile” devices, defined as those with publicly announced approvals.
In the US, devices are “approved” through premarket approval (PMA), humanitarian device exemption, and premarket supplement pathways, or “cleared” through the 510(k) process. For simplicity, we use “approval” to refer to both types of authorization, unless otherwise denoted.
Firstly, we searched Factiva (Dow Jones, New York, NY) for company press releases and news articles mentioning devices granted CE marking. We did a similar search through annual reports, financial regulatory filings (eg, with the US Securities and Exchange Commission), transcripts of calls on earnings and investor relations, and stock analyst reports using two commercial databases (S&P Capital IQ, McGraw Hill Financial, New York, NY, and Bloomberg, Bloomberg LP, New York, NY). We then consulted market research reports, trade publications, and search engines, and cross checked our results with a commercial database of marketed medical devices (Evaluate, London, UK). We also reviewed companies’ websites and marketing materials to match product names, which might differ between the US and EU. Finally, we supplemented these searches with a manual review of the FDA review summaries for PMA devices approved between 1 January 2005 and 31 January 2016; these summaries typically include a “prior marketing history” section. We initially downloaded all data on 30 May 2015 and subsequently updated the data through 31 January 2016.
Data extraction and coding
We extracted key information, including product type (eg, implantable electric stimulator) and trade names (eg, Brio Neurostimulation System), therapeutic area and subcategory (see supplementary appendix table 1⇑), sponsor’s name, marketing status in the US and EU, and date of CE marking. We also matched devices approved in the EU to the FDA’s publicly available lists of devices that have been cleared or approved for marketing in the US through the 510(k), PMA, premarket supplement, or humanitarian use pathways. To identify devices that may have been later withdrawn or discontinued, we cross referenced public sources and investor materials for any mention of marketing or sales in the US. For devices also marketed in the US, we extracted the date of clearance or approval, as well as the applicable regulatory pathway, and identified whether devices were approved first in the US or in the EU. We categorized the sponsors of devices as small companies if they had gross revenues of less than $1bn at the time of device approval. We searched Medline, Embase, and Web of Science for peer reviewed publications of the pivotal clinical trials used to support regulatory approval or used to demonstrate safety and effectiveness of the device, or both. We excluded short case series reports (<25 patients).
Using a prespecified protocol (see supplementary appendix), we categorized the novelty of each device in our sample as a “major innovation” or an “other change.” A major innovation was defined as a device that: was the first in a new class of products; involved new technology and made new claims with respect to its safety or effectiveness, or both; or involved new technology and was intended to be used in a new or expanded patient population. For example, the Melody Transcatheter Pulmonary Valve was categorized as a major innovation because it was the first transcatheter pulmonary valve that received regulatory approval in the US or EU (CE marking granted in 2006). “Other change” devices included products that had already been marketed, as well as devices that differed from existing products owing to minor technical, mechanical, system, or manufacturing changes.
Finally, we determined if each device in the cohort had been the subject of a recall or a post-marketing safety alert. We searched for recalls, field safety notices (which are issued when a device needs to be recalled for technical or clinical reasons), and other safety related corrective actions (eg, software updates to prevent possible adverse patient outcomes or device malfunction) using publicly available databases maintained by the FDA and national regulators in the two largest medical device markets in Europe: Germany (Bundesinstitut für Arzneimittel und Medizinprodukte) and the United Kingdom (Medicines and Healthcare products Regulatory Agency). We excluded recalls and safety alerts that were limited in scope (eg, a safety alert for a small number of lots because they had not been properly sterilized during manufacturing) or that were unlikely to cause adverse health consequences or technical failure (eg, a field safety notice for misspellings in the product label). To be conservative, we also excluded manufacturing related recalls for three products (Confient ICD and Livian CRT-D (both by Boston Scientific, MA, USA) in 201029; iDuo System (ConforMIS; MA, USA) in 201530), for which the manufacturers stated that patients were unlikely to be harmed (including these recalls in our analysis strengthened our results).
We calculated the unadjusted median difference in time between approval in the EU (CE marking) and in the US (approval or clearance by the FDA), rate and relative risk of safety alerts and recalls, and median time to publication of the pivotal study. Time to first safety alert and recall as well as time to publication were calculated from the time of first approval.
In adjusted analyses, we constructed: multivariable linear regression models to examine factors associated with greater time differentials in approvals between the EU and the US; Cox proportional hazards regression models to examine factors associated with safety alerts and recalls; and Cox proportional hazards regression models to examine factors associated with publication of trial outcomes for devices categorized as major innovations. Models included all variables of interest regardless of statistical significance: therapeutic subcategory, regulatory pathway, size of sponsoring company, an indicator variable for devices categorized as major innovations (except for models predicting publication), an indicator variable for devices approved first in the EU (which included devices not yet approved in the US), and an indicator variable for devices approved only in the EU. To account for secular trends over time, we included indicator variables for year of CE marking or year of first approval for the linear and Cox regression models, respectively. In sensitivity analyses, we repeated our analysis of safety events using: only recalls, a continuous time variable instead of an indicator variable for approval year, indicator variables for therapeutic area, and subcategory. We also repeated our analyses using logistic regression models and obtained substantively similar results.
Statistical analyses were performed using Stata version 12 (StataCorp). We considered two tailed P values less than 0.05 to be statistically significant.
No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for design or implementation of the study. No patients were asked to advise on interpretation or writing up of results. There are no plans to involve patients in the dissemination of results, nor will we disseminate results directly to patients, beyond our general media communications plan.
Overall, 309 new devices were identified that received Conformité Européenne (CE) marking between 2005 and 2010: 245 (79%) cardiovascular, 28 (9%) neurologic, and 36 (12%) orthopedic devices (table 2⇓). Nearly one quarter (75 of 309, 24%) was classified as major innovations. Most were developed by large companies (174 of 309, 56%).
Regulatory approval in the EU and US
The majority of devices approved for use in the European Union in our sample were also approved for use in the United States (206 of 309, 67%), of which most were approved first in the EU (129 of 206, 63%). Among the devices approved in the EU and US, 44 (21%) were approved through the premarketing approval (PMA) pathway, 76 (37%) approved as supplement PMA, 82 (40%) cleared through the 510(k) pathway, and 4 (2%) approved as humanitarian use devices. Of the 75 devices categorized as major innovations, a greater proportion was approved in the US (54 of 75, 72%), with 20 (27%) approved through the PMA pathway in the US. Among the 54 devices categorized as major innovations and approved in the EU and US, most were approved first in the EU (40 of 54, 74%).
The median time difference between CE marking and FDA approval was one month (FDA clearance before CE marking) for 510(k) clearance, 11.0 months (CE marking before FDA approval) for PMA supplement, 12.3 months (CE marking before FDA exemption) for humanitarian exemption, and 36.3 months (CE marking before FDA approval) for PMA approval (P<0.001 for comparison across categories) (fig 2⇓). In multivariable linear regression modeling adjusting for therapeutic subcategory, level of innovation, size of sponsoring company, and year, PMA and PMA supplement devices were associated with time differences of 38.0 (95% confidence interval 30.0 to 46.1; P<0.001) and 20.5 (12.5 to 28.5; P<0.001) months, respectively, between CE marking and subsequent FDA approval (see supplementary appendix table 2⇑).
Analysis of safety alerts and device recalls
Overall, 73 devices (24%) in the study cohort were subject to a recall or post-marketing safety alert as of 31 January 2016. Examples of safety events included the recall of an automated biventricular support system owing to a defect that could cause the device to shut down and stop pumping without warning (Abiomed AB-5000 Console, 2010), and a safety alert and labeling change for a neurostimulator, warning that reducing or ceasing stimulation could cause worsening of seizure frequency or severity compared with before implantation (Medtronic Deep Brain Stimulation, 2015).
The unadjusted rate of safety alerts and recalls in devices approved first in the EU (including CE marked devices not yet approved in the US) was 27% (62 of 232) compared with 14% (11 of 77) for devices approved first in the US (risk ratio 2.1, 95% confidence interval 1.0 to 3.4; P=0.04). In multivariable Cox regression modeling, devices approved first in the EU were associated with a 2.9-fold greater rate of safety alerts and recalls (hazard ratio 2.9, 95% confidence interval 1.4 to 6.2; P=0.005) and a 4.6-fold greater rate of recalls (4.6, 1.5 to 14.0; P=0.006) than devices approved first in the US (fig 3⇓ and supplementary appendix table 3). Similar results were obtained in sensitivity analyses using a continuous time variable and adjustment for therapeutic area and subcategory (see supplementary appendix tables 4 and 5). For example, a drug eluting stent received CE marking in 2006. However, a year later the stent was recalled after results of a US pivotal trial showed that the device was associated with significantly increased rates of major adverse cardiac events, including myocardial infarction and target vessel revascularization.31
Peer reviewed publication of primary trials of device safety and effectiveness
Among the 75 novel devices categorized as major innovations, the results of pivotal trials were published for 37 (49%). The median time to publication from first regulatory approval in the US or EU was approximately 3 years (37 months; range 0-118 months). Overall, the rates of publication one year, two years, and five years after regulatory approval were 7%, 17%, and 37%, respectively. In multivariable Cox regression modeling, devices approved in the US through the PMA pathway were significantly more likely to achieve publication than non-PMA approved devices (hazard ratio 8.6, 2.8 to 26.9; P<0.001).
We found that among high profile cardiovascular, neurologic, and orthopedic devices authorized for approval in the EU between 2005 and 2010, including those categorized as major innovations, the majority were approved first in the European Union and subsequently in the United States. Overall, a quarter of the devices in our study were associated with safety issues after they reached the market, ranging from communications about expanded safety warnings to recalls. Devices approved first in the EU were also associated with a nearly threefold greater rate of safety alerts and recalls.
Comparison with other studies
In a study of Food and Drug Administration approved prescription drugs, Carpenter and colleagues found that drugs approved closer to a regulatory deadline were more likely than other drugs to have serious safety issues, including withdrawal from the market for safety reasons and black box warnings.32 In a review of urgent recalls of devices (those with a reasonable chance of causing serious injury or death) issued by the FDA, Zuckerman and colleagues found that between 2005 and 2009 these recalls related to 35 cardiovascular devices, including 12 devices approved through the PMA pathway.26 During this period, the FDA approved 45 new cardiovascular devices through the PMA pathway,33 implying a rate of safety issues (12 of 45, 27%) comparable to our estimate. Similarly, Day and colleagues examined rates of recall of FDA approved orthopedic devices and found that such devices cleared through 510(k) were 11.5 times more likely to be recalled for any reason than were PMA approved devices (17.8% v 1.6%).34
Evaluating safety problems associated with devices in the US and EU has previously occurred mostly through anecdotal reports and uncontrolled analyses. A case series report by the FDA warned “the limited testing required in the EU can fail to predict dangerous risks and lack of effectiveness in actual use” and documented 12 high risk devices approved only in the EU that were “ultimately withdrawn from the market, but only after thousands of patients were harmed.”8 One industry sponsored analysis found that the absolute number of serious device recalls did not differ between the US and EU, but the relative rate of all safety issues was not determined.35
The median delay of three years between Conformité Européenne (CE) marking and US approval for PMA pathway devices—which are usually the highest risk implantable devices—in our study is consistent with prior industry estimates during a comparable period.36 Most of the medical devices (67%) in our study were approved in both the US and the EU, similar to the proportion of new prescription drugs approved in both locations (66%).37
Roughly half (51%) of the primary trials assessing the safety and effectiveness of devices judged to be major innovations remained unpublished or not conducted over five years after first approval, comparable to the rate reported in a study23 of high risk cardiovascular devices approved in the US between 2000 and 2010 (46 of 106, 43%). Similarly, Kynaston-Pearson and colleagues found that nearly half of recently introduced hip replacement prostheses had no published evidence of clinical effectiveness.38 By contrast, previous studies39 40 found that the results of 76-78% of pivotal trials for new drugs are eventually published. Information on devices approved in the EU, or pathways in the US that do not require premarket clinical testing, is even more limited.22 In a study of 10 devices reimbursed in Austria, Wild and colleagues found that evidence available at the time of CE marking are more often case series or small feasibility trials, compared with the controlled trials and large prospective cohort studies used as the basis for FDA approval.41 In turn, low rates of publication and lack of high quality evidence may adversely impact patient outcomes. Nieuwenhuijse and colleagues found that innovative and well known hip and knee replacement prostheses were widely implanted despite a lack of high quality evidence supporting their use; several of these devices were also associated with inferior survival and higher rates of revision.27
Limitations of this study
Our study has several limitations. Firstly, we identified devices approved in the EU based primarily on company announcements, investor reports, and news articles on marketed products. Since we focused on devices with publicly announced regulatory approvals, our findings might not be generalizable to devices that are authorized but not marketed or that are introduced without formal notice to the public or shareholders, and this study likely underestimates the total number of approvals during the study period. However, this cannot be definitively determined. In a prior investigation by The BMJ, the UK Medicines and Healthcare products Regulatory Agency was unable to provide any data on the numbers of high risk devices currently in use,42 and similarly denied this information to researchers.28 43 None the less, given the strong incentives for companies to market and sell their products as widely as possible,44 we believe our studied devices represent those most likely to be used by patients and clinicians.
Secondly, we focused on three disease areas, but these include the most commonly used implantable and high risk devices in clinical practice45 46 47 and represent approximately 70% of all high risk devices approved in the US.20 Thirdly, although we chose our study period to allow at least five years of follow-up (consistent with the upper bound of prior estimates of the time differential between CE marking and FDA approval35), with time, more of the devices currently approved only in the EU may be approved in the US, and more trials may be published. Fourthly, CE marking does not guarantee that patients will immediately have access to new products, and actual delays in access between the US and EU may be smaller after accounting for the time associated with reimbursement decisions by EU payers.48 Fifthly, there may be additional safety issues that have not yet triggered regulatory action, which would mean that our study underestimates the true prevalence of device related safety concerns. It is also possible that the standards for issuing safety alerts and recalls may differ across jurisdictions, although prior studies have reported that the absolute number of safety related announcements is comparable between the US and EU.34 To mitigate the potential for bias, we included all safety alerts and recalls issued in both the US and the EU. Finally, an additional number of devices may be ineffective, and other reasons may explain why certain devices approved in the EU are not also submitted for approval in the US. For example, in a large randomized controlled trial, a novel CE marked biodegradable drug eluting stent did not improve clinical outcomes compared with an older drug eluting stent; the stent remains available only in the EU.49
Conclusions and policy implications
Despite these limitations, this study provides an important empirical measure of the trade-offs associated with the US and EU frameworks for regulating medical devices. Although most devices, including the majority of those identified as major innovations, are ultimately approved in both the US and the EU, patients in the US may not have access for up to three years later than patients in the EU. Supporters of the current EU system argue these delays have public health implications if some patient populations can benefit from earlier treatment, although a device’s true benefits might only be known in retrospect after the trials needed for FDA approval are conducted. In the US, regulators could improve their efficiency and minimize delays by fast tracking approval of devices with the potential to substantially improve patient health outcomes. To that end, in 2014 the FDA announced a pilot program to expedite development and approval of certain high risk medical devices for serious or life threatening conditions characterized by unmet medical needs.50
However, the possible benefits of faster access must be carefully weighed against the risks arising from devices, as was observed with a number of drug eluting stents initially hailed as breakthroughs,51 but later found to be unsafe or ineffective.52 53 54 55 There has been growing consensus by patients, physicians, payers, and policymakers that the current EU system could be revised to better safeguard patient safety.56 57 58 59 60 61 62 63 64 65 66 67 Compared with the US, at the time of approval, patients in the EU have substantially less and lower quality information on the potential benefits and harms of new devices, which raises ethical concerns because CE marking may be misinterpreted as signifying that devices are safe and clinically effective.40 The European Society of Cardiology and others have called for a centralized system for evaluating high risk devices; stronger and more transparent clinical data requirements that incorporate expert medical advice; and public education on the limitations of CE marking.68
As of January 2016, however, the proposed revisions of the EU’s directives on medical devices did not include a new regulatory body focused on devices. At a minimum, policymakers should require greater transparency, including a public register of all CE marked devices and summaries of their regulatory decisions (as is the case for drugs in the EU and both drugs and devices in the US). The European Databank on Medical Devices (Eudamed) already collects some of these data, but this database is not publicly accessible. Such information is important for patients and clinicians to make informed decisions about treatment as well as for researchers, who might not know that similar products have been studied in unpublished trials.69 Beyond greater transparency, policymakers could consider compromise solutions that would benefit both patients and device manufacturers, such as coupling more rigorous evidentiary standards for the highest risk devices with a streamlined pathway for reimbursement across the EU, as well as conditional approval and funding of breakthrough technologies.70 Indeed, one economic modeling study found that between 2004 and 2013, coronary stents studied in clinical trials were more likely to be commercially successful than those that were not, and European utilization of new stents increased substantially after the results of clinical trials conducted for US approval were released; the authors concluded that the EU could meaningfully improve patient welfare by increasing the informational standard required for market access.71
Patients and clinicians need access to, and balanced presentation of, the available evidence of the safety and effectiveness of novel devices, as well as clear communication about the evidentiary gaps. Our findings suggest that products introduced earlier in their development cycle are also more likely to increase the risk of harms, underscoring the urgent need for transparency to make truly informed decisions.
What is already known on this topic
In the European Union, medical devices are approved by private notified bodies if they meet performance criteria and are likely to be safe, but notified bodies generally do not require evidence of effectiveness for most devices
Many high risk devices are approved faster in the EU than in the United States, where the Food and Drug Administration usually requires prospective clinical trials of such devices
Controversy over the safety of implanted devices approved without rigorous clinical testing has led to calls for regulatory reforms in the EU and US
What this study adds
Most high profile medical devices approved in the EU (devices for which the approvals are publicly announced), including devices identified as major innovations, are eventually also approved in the US
Medical devices available in the EU before the US are at higher risk for emergence of safety issues
Evidence of clinical performance or outcomes for many new medical devices that could guide treatment decisions remains unpublished or unavailable up to five years after approval
Contributors: All authors contributed to the conception and design of the study. TJH extracted the data. TJH and JMF did the statistical analyses. All authors contributed to interpretation of the results; read, wrote, and critically revised the manuscript for important intellectual content; and made an important contribution to the research and manuscript, and approved the final version for publication. TJH is the guarantor. The authors’ employers and funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Funding: This study received no specific financial support.
Competing interests: All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf and declare that (within the past three years): TJH has received funding from Harvard University and the Interfaculty Initiative in Health Policy and was employed by Blackstone and Bain Capital; JMF has received funding from the Patient-Centered Outcomes Research Institute and Merck, and has consulted for Aetion, a software company; and ASK is a Greenwall faculty scholar and has received funding from the Harvard program in therapeutic science, the Laura and John Arnold Foundation, and the FDA Office of Generic Drugs and Division of Health Communication.
Ethical approval: Not required.
Data sharing: No additional data available.
Transparency: The authors affirm that the manuscript is an honest, accurate, and transparent account of the study was reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.