Balancing the evidence for guidelines: lessons from the NICE abdominal aortic aneurysm guidance—an essay by Bruce CampbellBMJ 2020; 370 doi: https://doi.org/10.1136/bmj.m3480 (Published 24 September 2020) Cite this as: BMJ 2020;370:m3480
The National Institute for Health and Care Excellence recently published a guideline on the diagnosis and management of abdominal aortic aneurysms.1 This was the culmination of a long, uncomfortable period of conflict with much of the vascular community and indeed with its own advisory committee. Typically, the disagreement was about one very small but important element of the draft. NICE revised the committee’s recommendation in line with stakeholder feedback but has been criticised for doing so.2
This episode provides some useful insights into the challenges facing NICE and what is involved in producing “evidence based” guidance for health services.
A brief history of the NICE aortic aneurysm guidance
The conflict was about the use of endovascular aneurysm repair (EVAR), which has become more popular worldwide for treating abdominal aneurysms than open surgery.3 EVAR has much lower perioperative risk and quicker recovery, but it requires long term follow-up with imaging and many more re-interventions than open surgical repair. Prevention of rupture is the main reason for elective aneurysm repair, but large, well designed randomised controlled trials (RCTs) have shown that, in the long term, the survival of patients who have had EVAR might be lower than that of those who have had surgical repair.4 RCTs have also shown that the survival of patients unfit for open repair is no greater overall after EVAR than after no intervention.5
NICE’s guideline committee reviewed all these data and also considered extensive economic modelling, which showed that there were no circumstances in which EVAR was as cost effective as open surgery in the long term (up to 15 years). Based on these considerations, it drafted recommendations that EVAR should not be offered to patients suitable for open repair nor to patients unsuitable for open repair because of their general medical condition.
The vascular community was horrified—not just in the UK but around the world. These recommendations overturned years of progress towards a minimally invasive approach to treatment of aneurysms. Manufacturers, too, were deeply unhappy: stent grafts for EVAR had become an important source of income. The response to NICE’s consultation on its draft guidance strongly reflected these views, and NICE’s proposals were debated vitriolically at vascular meetings.
The draft recommendations had a sound basis in terms of the RCT evidence and cost modelling, but they gave little weight to some potentially important patient matters. Many vascular specialists, and some published evidence, have said that many patients with abdominal aortic aneurysms prefer EVAR: they do not want to face a very major operation if there is a viable alternative, the need for long term follow-up notwithstanding.6
Another more arcane issue is the anxiety and distress that patients might experience if they are denied any treatment, living with the knowledge that their aneurysm might rupture at any time, causing their death.7
Balancing the range of disparate evidence
These conflicting considerations exemplify the difficulty in balancing “hard evidence” (RCTs, economic modelling) against the “softer” evidence of patient experiences and preferences when making recommendations about the use of interventions. NICE committees have a duty to take patient issues into consideration when making their recommendations. Many years ago, NICE developed its Social Value Judgements, now subsumed by NICE’s Principles, which discuss the topics of individual choice and societal benefit. What weight to give to these aspects, compared with the findings of RCTs and economic analysis, is a dilemma. In the case of the aneurysm guidance, the committee remained convinced that RCT results and the economic evidence dominated.
The contributions of patients and carers can help interpret aspects of the scientific literature. Often, they give reassurance by supporting the main conclusions and messages that the published clinical evidence provides.8 The problem arises when patient perceptions and preferences conflict with the hard evidence of clinical trials and cost effectiveness. How can these different types of evidence best be balanced against each other, and what type and volume of patient and public input is most useful?
Patient input—which outcomes matter
These aspects continue to tax those involved in health technology assessment and health policy.9 When seeking hard evidence, there are well established systems for searching the published literature, selecting the most relevant and cogent studies, and grading the strength of the evidence that they provide. By comparison, getting and gauging evidence from patients, carers, and the public is an uncertain art. People can contribute as individuals or as representatives of patient groups. Their input might be based on personal experience or concern for others. They may be involved through written submissions, interviews, focus groups, attending decision making committees, and increasingly through a range of online avenues, including public consultation on draft recommendations and guidance.1011
The contributions of individuals and patient organisations to decision making is just one (important) part of the complex matrix of balancing patient related perspectives against the hard endpoints of clinical research and cost effectiveness. Because quality of life measures form part of the calculation of “cost per QALY” (quality adjusted life year—usually measured via validated methods, in clinical studies), the effects on patients’ physical and mental health are incorporated into that parameter of decision making. But some aspects of patient preferences might not be captured in a cost per QALY approach—such as the desire to avoid the early impact and risks of open surgical treatment and the potential mental consequences of living with an untreated aortic aneurysm.
In the case of the aortic aneurysm guidance, there was the additional influence of quite strong clinician preference for EVAR, but that was based in large part on the perception of clinicians that EVAR was the intervention that many patients preferred. There is no way of knowing, but good reason to suspect, that patients who are keen to avoid the early effects of open aneurysm surgery would not change their view on the basis of a slightly higher chance of death 8-15 years later, which (together with cost effectiveness modelling) was the main hard endpoint that was balanced against the soft patient centred considerations.
Big differences in the amount of evidence available
NICE’s advisory committees vary a lot in terms of the volume and quality of evidence that they expect and the amount of published evidence on which their decisions are typically based.
The interventional procedures and medical technologies advisory committees, for example, commonly have little high quality evidence on which to base their decisions because the procedures and devices they see are so new. Under these circumstances, other sources of evidence, including case series, registry data, and audits can be used, and the contributions of clinical experts and of patients and carers, will weigh more heavily in the balance than when RCTs are available. Considerations such as “promise” might also influence this kind of early decision making.
The technology appraisals committees see many new medicines, but with an enforced level of clinical trial data, which is mandatory for pharmaceutical products. NICE guideline committees benefit from the output of all these other groups and commonly have access to major clinical trials for much of the care that they recommend (as was the case for the aortic aneurysm guideline).
Attention to powerful patient advocacy is not without risks, especially in the context of longer term outcomes.
In 2014 the NICE medical technologies programme considered MAGEC, a system for spine lengthening in children with scoliosis. The MAGEC implant consisted of titanium rods, with an internal magnetic distraction system, which allowed intermittent lengthening of the spine using an external power unit. This avoided the repeated open surgical operations required to lengthen conventional spinal rod systems.
Some good clinical trial evidence indicated that the system worked well. In addition, the advisory committee heard from the parents of children who had had a conventional spinal rod system replaced by a MAGEC device. They said that their children’s lives had been transformed. They no longer dreaded hospital admissions because of what was involved in repeated open surgery. Instead they had a simple outpatient visit in which they were distracted until they felt a twinge of pain and the procedure was stopped.
Based on this concurrence of short and medium term published evidence and strong patient and parent advocacy, in 2014 NICE recommended that the MAGEC system be considered for selected children with scoliosis, based on both clinical and cost consequence considerations.12 In 2017, however, safety concerns were expressed both by regulators and in the media, owing to separation of a component of the MAGEC system, with subsequent corrosion of internal parts and the production of metallic debris, such that the systems needed to be removed. In 2020, supply of all MAGEC systems was suspended in the UK.
The 2014 NICE recommendation would likely have been similar even without the persuasive testimony of the parents of affected children, but they were an important contribution to the debate. This example emphasises the great importance of continuing to collect and review “real world” data on implants and other devices, even after clinical trials have been sufficient for them to be approved and recommended for use and patients and carers have lent strong support.
The final arbiter
Guidance is not published by independent advisory committees but by NICE itself. This is true for official organisations producing healthcare guidance around the world. When NICE receives draft guidance from any of its advisory committees, it is bound to take account of the public consultation responses that have been received. When the response to public consultation has been voluminous, controversial, or hostile, then NICE might hold meetings with important groups of consultees, such as vascular specialty representatives in the case of the aneurysm draft.
NICE needs to take account of the over-riding principle of “reasonableness,” which underpins all NICE decisions. If NICE is persuaded that the advice it has received from one of its advisory committees has not achieved the best balance, then it is entitled to make changes—in this case by adopting a more flexible approach to the use of EVAR.1
By coincidence, this flexible approach to EVAR is in tune with the pressures of the continuing covid-19 pandemic, during which there has been a gross reduction in the volume of vascular procedures. Instead EVAR has been used whenever possible rather than open surgery, and this might persist.
Like any controversy around NICE’s work, the long term fallout helps to steer clinical practice. In the case of aneurysm treatment, it has prompted careful reconsideration of the extent to which EVAR is used. It has shown the very wide range of considerations that come into play in producing healthcare guidance. In particular, it has drawn attention to the great difficulty in balancing the hard evidence of clinical trials and cost modelling with the many aspects of patient preferences. Involving patients, carers, and the public in decision making on healthcare policy goes a long way towards reaching the “best” decisions, but it cannot solve the dilemma of how much weight to give to all the different types of evidence that might come into play.
Bruce Campbell is a vascular surgeon and has had prominent roles in the Vascular Society of Great Britain and Ireland, including honorary secretary. He has published widely on vascular surgical topics, including provision of vascular services. He has also had considerable involvement with NICE—as a member of its technology appraisal committee (2001-03) and as chair of its interventional procedures (2002-15) and medical technologies (2009-15) advisory committees. He has led studies and published on NICE methods, including patient and public involvement.
Competing interests: I chaired standing NICE advisory committees from 2002 to 2015, and have published on NICE methods and on HTA in general. I am also a well known vascular surgeon. I have published a study on patient preferences with regard to treatment of abdominal aortic aneurysms, in 2009. I had no formal involvement in the production of the NICE guideline on management of abdominal aortic aneurysms.
Provenance and peer review: Commissioned; not externally peer reviewed.