IDEAL-D: a rational framework for evaluating and regulating the use of medical devicesBMJ 2016; 353 doi: https://doi.org/10.1136/bmj.i2372 (Published 09 June 2016) Cite this as: BMJ 2016;353:i2372
- Art Sedrakyan, professor1,
- Bruce Campbell, professor2,
- Jose G Merino, clinical research editor3,
- Richard Kuntz, chief scientific, clinical, and regulatory officer4,
- Allison Hirst, researcher5,
- Peter McCulloch, professor5
- 1Department of Healthcare Policy and Research and Medical Device Epidemiology Network (MDEpiNet) Science and Infrastructure Center, Weill Medical College of Cornell University, New York, NY, USA
- 2Interventional Procedures Programme, National Institute for Health and Care Excellence, London, UK
- 3The BMJ and Johns Hopkins Community Physicians, Bethesda, MD, USA
- 4Medtronic, Minneapolis, MN, USA
- 5Nuffield Department of Surgical Science, University of Oxford, Oxford, UK
- Correspondence to: P McCulloch
- Accepted 29 March 2016
Implantable devices such as pacemakers and hip implants have transformed many lives, but there have also been high profile instances of harm.1 2 3 4 5 Unlike the system for drugs, marketing approval for devices in the European Union and the United States has historically focused on proof of safety as a minimum requirement, and approval could be granted based on preclinical evidence alone, with no randomised clinical trials (fig 1⇓). In both jurisdictions, recent years have seen some tightening of requirements. In the US, more invasive devices now generally require a rigorous “pivotal” clinical trial either through the FDA’s pre-market approval (PMA) pathway or within the 510k pathway. In the EU, the CE mark requirements are being moved in a similar direction by the evolving medical device reform (MDR) programme.
Despite these improvements, both systems dichotomise device status as either pre-market (not yet approved) or post-market (approved), so removing incentives for fuller evaluation and reporting at both early and later stages in the evolution of the device. A system in which data requirements for device approval and surveillance can be matched to the device’s stage of development, known as total product life cycle evaluation,6 would encourage better evidence development and reporting both before and after a pivotal trial, in much the same way that the Kefauver-Harris Amendment 1962 did for medicines after thalidomide.7
While the total product life cycle idea is attractive in principle, there has been little guidance on implementation. The IDEAL (Idea, Development, Exploration, Assessment, Long term study) framework and recommendations,8 a model for integrated stepwise evaluation of maturing interventions, has already been introduced for developing and evaluating new surgical procedures. We discuss IDEAL’s potential to provide a structure for evaluation and regulation of devices and how a partnership between surgeons, academics, and regulators could facilitate such a move without major new resources.
What is IDEAL?
The IDEAL framework was developed by an expert consensus group to describe what types of studies and reporting should be used for new surgical procedures, from first use through to adoption in practice (table 1⇓).9 The initial idea stage focuses on the “first-in-human” uses of new treatments, processes, or procedures. In the subsequent development stage inventors modify the technique, often through a fairly rapid sequence of iterative improvements. In the exploration stage other investigators become involved, and technical details, indications, operator learning curves, and quality control are discussed. In the assessment stage operators collaborate on a definitive study of the new technique. Lastly, long term study is needed to detect late and rare side effects, “indication creep,” and performance variation. The IDEAL recommendations are designed to address the specific questions that arise at each stage.
How could IDEAL be used for medical devices?
Although many aspects of IDEAL are easily applied to invasive therapeutic devices, there are some differences between devices and surgical procedures that would necessitate some modifications to the framework.
Need for stage 0
In surgery most of the iterative modification of the procedure occurs in stage 2a (development), but with devices these modifications mostly occur before the device is first used in humans (table 2⇓). For medical devices, therefore, the IDEAL framework should begin during preclinical development, when product design, materials, and functional components are determined and tested to prepare the device for first-in-human (stage 1) studies. This initial preclinical stage would logically be designated stage 0 in an “IDEAL-D” (devices) system and can be important in surgery too: animal models are often used to optimise new valve repair techniques before human studies begin.10 11 There are many reporting guidelines for different kinds of preclinical research, but international minimum reporting standards for studies of therapeutic devices are lacking.12 The traditional preclinical tests for hip implants, which were unlikely to identify key issues affecting wear of metal-on-metal devices, show the inadequacies of current standards.13
The principles for stage 0 reporting will have to balance the need to protect intellectual property against proof that appropriate ex-vivo tests of safety and reliability have been conducted. An IDEAL-D recommendation for this stage (initially only for implantable devices) needs to have consensus between scientists and engineers, regulators, and industry representatives.
Stage 1: no change, but legislation needed
The current IDEAL recommendations start at stage 1 by suggesting that all first-in-human studies should be internationally registered and a functionally useful description made available to others. Universal registration of studies could allow identification of benefit and harm themes and trends by surveillance of incoming reports and detect possible device harms at an early stage, ensuring that unsuccessful innovation is not repeated through ignorance. Ultimately, innovators could be ethically obliged to search the registry before embarking on a first-in-human study to avoid repeating a harmful error reported by another investigator, just as some research funders already require a systematic literature review before submission of major grant applications to avoid research waste. One thorny issue, however, is how to protect reports of device innovations that proved harmful from being used for legal challenges. Precedents exist within the transport and energy industries for legal firewalls for reporting incident investigations and could be used as models for device regulation.14 15 Legislation would undoubtedly be required at US Congressional or EU levels.
Stages 2a and 2b: fuse or keep separate?
The recommended study formats for IDEAL stages 2a and 2b were designed with specific surgical situations in mind. Prospective development studies, recommended in stage 2a, cater for the “tinkering” stage of rapid iterative innovation in surgical procedures, while prospective exploration studies for stage 2b have the important objective of establishing a consensus among surgical teams. By contrast, most technical device iterations occur during stages 0-1 and involve a single manufacturer. A single stage 2 might therefore suffice, aimed, as with the current prospective exploration studies in stage 2b, at facilitating progression to definitive randomised controlled trials in stage 3. Having a regulatory requirement for a stage 2 study would considerably enhance the average level of clinical evidence supporting new devices and would “set up” such devices for a definitive study.
Stage 3: mandatory randomised controlled trials for devices?
A controversial question is whether regulators or purchasers of healthcare should require randomised controlled trials showing clinical efficacy for devices that are similar to existing devices (“me too” products), as they do for drugs. From an ethical viewpoint, the evidence that inadvertent patient harm can result from apparently small modifications to existing devices suggests that a clinical trial should normally be required for most implantable devices.16 17 However, the range of medical devices is wide, with most posing much lower risk than implantable devices. Practical considerations are likely to lead regulators and policy makers to define a limited category of devices for which stage 2 studies and stage 3 randomised controlled trials will be required. IDEAL allows for valid experimental designs other than randomised controlled trials in stage 3, and innovative ideas such as tracker trials, adaptive designs, and studies based on economic modelling might become attractive options, allowing inclusion of incremental innovations of the device and other device comparators.18 19
Reporting of health economic data for new devices is also important, and would be relevant in stage 3 or late in stage 2. However, some experts resist the idea of including health economics in the IDEAL framework,20 and this will require further multidisciplinary debate.
Stage 4: registries to begin earlier?
IDEAL recommends registry designs in stage 4 (long term follow-up), but opportunities for device evaluations in registries exist at all stages and will soon increase, as unique device identifiers become available and are implemented in registries and related data systems.21 22 If prospective registries were started from the outset of clinical use they could be used continuously for safety surveillance of higher risk devices, picking up weak signals from rare events and outlier device reports as technology proliferates. Hence, surveillance by registries from first-in-human experience onward would be a valuable component of an IDEAL-D system. It would allow opportunities for “nested” randomised controlled trials23 and for using risk adjustment techniques to analyse large registry datasets to study small or long term effects in situations with multiple confounders, in which a randomised trial might be infeasible.24
Implications of applying IDEAL-D recommendations to regulatory evidence
Policy makers are likely to be pressured to make exceptions to any rules developed to ensure that innovative devices are subjected to proper evaluation. Special cases may exist for devices used as part of responses to natural disasters or public health emergencies, when formal scientific testing along the whole IDEAL pathway may not be feasible. In such situations the best available evidence should still be collected using evaluation appropriate to the earlier stages of the IDEAL framework.
First-in-kind devices (breakthrough innovations) for incurable or high risk patients show the value of IDEAL-D. It seems reasonable that highly innovative devices should normally be subjected to the whole integrated pathway of IDEAL stage studies, including prospective development and prospective exploration studies, randomised controlled trials, and registry studies, but some argue that patients with no chance of survival without treatment should be given access to potentially lifesaving treatment without a randomised comparison (in the context of stages 2a and 2b studies). A recent relevant example has been the use of transcatheter aortic valve replacement for patients who were not candidates for surgery. However, as technology starts replacing traditional valve surgery, the data from such uncontrolled studies may lead to recognition that a randomised trial is needed—for example, because of doubts over treatment durability. Having an initial non-randomised phase can also allow the collection of crucial data on learning curves so that a trial can proceed without operator bias.25 26
Where a new implantable device takes a completely novel approach a randomised controlled trial in stage 3 seems justified, but controversy will arise when competitors make incremental changes to a basic design, not seeking to substantially change the mechanisms of action but to compete on other grounds, such as cost. If regulators insist on a randomised controlled trial to demonstrate equivalence for such devices, this may better protect patients against unexpected harm but would change the commercial playing field by placing a major evidence barrier in the way of general marketing. If novel (first-in-kind) devices were put into a register when they reached IDEAL stage 4, subsequent similar devices could be required to join the register. This might provide the infrastructure to conduct subsequent randomised controlled trials comparing the new competitors with the first device, using the “trials within cohorts” or “nested within registry” design approach.22 23
Devices that are already marketed and have become established in the absence of rigorous evaluation—for example, robotic surgery or hip and knee replacement devices—pose a problem for regulators who want to ensure fairness in applying standards of evidence. Where these are already in common use, stage 4 registries may be the only enforceable solution for evaluation and should be mandatory for implants and devices that enable major surgery. Evidence emerging from such registries may highlight the need for more detailed studies of specific devices or clinical groups.
Different types of registry may be needed for novel devices and for “legacy” devices that have already been in use without rigorous evidence. Novel implantable devices—for example, transcatheter cardiac valves—could progress through IDEAL stages 2 (and usually 3) before a detailed, product specific registry is developed, mandated by the regulator. For legacy devices—for example, robotic surgery—a simpler population registry covering all manufacturers and varieties may be needed, similar to international joint replacement registries.27 These would collect small minimum datasets restricted to measures needed for evaluation of care. Funding and curatorship of such registries would need to be agreed between the government, industry, and clinicians to allow them to be effective.
We need a safe regulatory system that does not unnecessarily burden innovation but encourages the production of adequate valid evidence on safety and efficacy. IDEAL has therefore partnered with the Medical Device Epidemiology Network, a US public-private partnership that aims to develop more rigorous post-market surveillance systems and integrate these better with the approval process. Since the 2011 incubator think tank that kick started this collaboration, two further seminars involving regulators, payers, surgeons, and patients have been held.28 The convergence of thinking has been evident in Food and Drug Administration guidance documents and IDEAL publications.29 30 Examples of initiatives that could implement the IDEAL recommendations are also emerging in the United Kingdom. “Beyond Compliance” is a collaboration between industry, clinical registries, and regulators to evaluate joint replacement devices from their earliest introduction to the market.31 IDEAL-D could provide a coherent, overarching framework for such initiatives.
Safer, faster access
An IDEAL-D framework for devices represents a potential paradigm shift in regulatory evidence. The current system, in which rigorous evaluation occurs mainly before devices reach the market, encourages delayed market release. A continuous evaluation process using IDEAL-D, however, could allow graded, responsible, but earlier patient access. This proposal offers the potential for better, safer, and faster regulatory evidence gathering for new devices. To achieve its full potential for improved surveillance, appropriate rationalisation of regulatory structures and procedures will be needed to minimise regulatory “drag”, particularly for early stage clinical studies.
An IDEAL-D based system could enable patient access to devices with adequate clinical evidence but without excessive delay or resource use. Cooperative registries would allow devices to be monitored worldwide throughout their life cycle. The IDEAL-D concept offers a suitable model for a new integrated evaluation pathway for regulatory science. We suggest that regulatory bodies internationally should seize the opportunity to achieve a greater degree of global unity by considering how they could harmonise their systems using IDEAL-D as the framework for evidence gathering.
Current evidence requirements for regulation and large scale purchasing of invasive therapeutic devices are inadequate to ensure safety and efficacy
A “total product life cycle” (TPLC) system of graduated approval as evidence accumulates would improve data collection at all stages in the evolution of a therapeutic intervention
The IDEAL framework and recommendations, developed for surgery, could easily be adapted to provide a practical template for a TPLC approach to regulatory evidence
Granting approval for device use or purchasing only in the context of appropriate IDEAL study types could allow graded, responsible patient access and accelerate evaluation while increasing its quality
Contributors and sources: All authors took part in a discussion forum at the Cornell Club, New York, in April 2014, organised by Professor Sedrakyan on behalf of the IDEAL collaboration, and volunteered to write a paper together to summarise the issues and debate. PM wrote the first and AS the final draft of the paper. All authors contributed by reading drafts and suggesting changes that were considered by the authors as a group.
Competing interests: We have read and understood BMJ policy on declaration of interests and declare that PM has received consultancy payments from the National Engineering Research Centre for Ultrasound Medicine, People’s Republic of China. All of these monies have been used to support the IDEAL collaboration. PM and AH have received support for travel and subsistence in relation to meetings about IDEAL. The IDEAL collaboration has received support from Medtronic, Stryker, Zimmer, Johnson and Johnson, the US Food and Drug Administration, the Health Technology Assessment Programme (UK), the Health Foundation (UK), and the Oxford Academic Health Science Network, all to support meetings.
Provenance and peer review: Commissioned; externally peer reviewed.