Use of magnetic resonance angiography to select candidates with recently symptomatic carotid stenosis for surgery: systematic reviewBMJ 2002; 324 doi: http://dx.doi.org/10.1136/bmj.324.7331.198 (Published 26 January 2002) Cite this as: BMJ 2002;324:198
- Marie E Westwood, research fellowa,
- Steven Kelly, research fellowa,
- Elizabeth Berry (), senior lecturer in medical physicsa,
- John M Bamford, consultant neurologistd,
- Michael J Gough, consultant vascular surgeonb,
- C Mark Airey, senior research fellow in public healthe,
- James F M Meaney, consultant radiologistc,
- Linda M Davies, senior research fellow in health economicsf,
- Jane Cullingworth, superintendent radiographerc,
- Michael A Smith, professor of medical physicsa
- a Academic Unit of Medical Physics and Centre of Medical Imaging Research, University of Leeds, Leeds General Infirmary, Leeds LS1 3EX,
- b Vascular Surgery, Leeds Teaching Hospitals NHS Trust, Leeds General Infirmary,
- c Department of Radiology, Leeds Teaching Hospitals NHS Trust, Leeds General Infirmary,
- d Department of Neurology, Leeds Teaching Hospitals NHS Trust, St James's Hospital, Leeds LS9 7TF,
- e Division of Public Health, Nuffield Institute for Health, University of Leeds, Leeds LS2 9PL,
- f Centre for Health Economics, University of York, York YO1 5DD
- Correspondence to: E Berry
- Accepted 12 October 2001
Objective: To determine if sufficient evidence exists to support the use of magnetic resonance angiography as a means of selecting patients with recently symptomatic high grade carotid stenosis for surgery.
Design: Systematic review of published research on the diagnostic performance of magnetic resonance angiography, 1990-9.
Main outcome measures: Performance characteristics of diagnostic test.
Results: 126 potentially relevant articles were identified, but many articles failed to examine the performance of magnetic resonance angiography as a diagnostic test at the surgical decision thresholds used in major clinical trials on endarterectomy. 26 articles were included in a meta-analysis that showed a maximal joint sensitivity and specificity of 99% (95% confidence interval 98% to 100%) for identifying 70-99% stenosis and 90% (81% to 99%) for identifying 50-99% stenosis. Only four articles evaluated contrast enhanced magnetic resonance angiography.
Conclusions: Magnetic resonance angiography is accurate for selecting patients for carotid endarterectomy at the surgical decision thresholds established in the major endarterectomy trials, but the evidence is not very robust because of the heterogeneity of the studies included. Research is needed to determine the diagnostic performance of the most recent developments in magnetic resonance angiography, including contrast enhanced techniques, as well as to assess the impact of magnetic resonance angiography on surgical decision making and outcomes.
What is already known on this topic
What is already known on this topic Carotid endarterectomy for recently symptomatic carotid stenosis is beneficial in patients with 70-99% stenosis as measured by conventional angiography
It is not known whether the less invasive imaging technique of magnetic resonance angiography can accurately identify patients who will benefit from surgery
What this study adds
What this study adds Magnetic resonance angiography is highly sensitive and specific in diagnosing 70-99% carotid stenosis
However, the studies on which this conclusion is based are of low quality and high heterogeneity
Patients with arteriosclerosis, particularly those with recent carotid territory stroke or transient ischaemic attack who might benefit from carotid endarterectomy, may be investigated with conventional angiography, ultrasonography, or magnetic resonance angiography. 1 2 Magnetic resonance angiography was introduced in the 1980s. The technology continues to evolve rapidly, and three main approaches are currently used: two dimensional time of flight methods, three dimensional time of flight methods, and three dimensional contrast enhanced techniques. Time of flight (or “inflow”) methods are non-invasive and emphasise flowing blood because a different signal is given by material that has moved into the imaging volume. Contrast enhanced methods use gadolinium based contrast materials that are better tolerated than the iodinated media used for conventional angiography. Contrast enhancement overcomes some of the drawbacks of time of flight techniques, especially signal loss near stenoses. A greater volume of the body can be imaged in a shorter time, allowing evaluation of the vasculature from the aortic arch to the circle of Willis.
The use of magnetic resonance angiography and ultrasonography has expanded rapidly in recent years, and the use of conventional angiography has declined. Although conventional catheter angiography remains the definitive imaging technique, it is an invasive procedure for which the patient must be admitted to hospital, it involves the use of ionising radiation, and when used in the carotid circulation it is accompanied by serious complications, including a 0.5-2% risk of stroke.3 Magnetic resonance angiography may be more acceptable to patients and may be of particular use in patients not suitable for conventional angiography—for example, patients with an allergy to iodinated contrast medium, frail and elderly patients, and patients with severe peripheral vascular disease. These potential benefits may be offset by poor performance as a diagnostic test.
This systematic review examines the evidence on the performance of magnetic resonance angiography in evaluating patients with recently symptomatic internal carotid artery stenosis. The North American symptomatic carotid endarterectomy trial (NASCET) and the European carotid surgery trial (ECST) found a clear benefit of surgery in patients with recently symptomatic stenoses of 70-99% as measured by conventional angiography with the NASCET criteria. 4 5 We therefore evaluated the evidence on the diagnostic performance of magnetic resonance angiography in comparison with conventional angiography at this threshold. NASCET and ECST found a smaller benefit of surgery in patients with symptomatic 50-99% stenosis; ECST also showed a clear downward trend in the benefit of surgery for stenoses less than 70%.5 The benefit in this group also depended on the age and sex of the patients. However, a recent Cochrane review concluded that surgery was beneficial for patients with 50-69% stenosis,6 so we also evaluated the evidence on the diagnostic performance of magnetic resonance angiography when these more moderate stenoses were included (that is, for patients with 50-99% stenosis).
For each diagnostic threshold, we aimed to answer the question “What are the sensitivity and specificity of magnetic resonance angiography, in comparison with the gold standard of conventional angiography, in distinguishing severely stenosed arteries suitable for surgery, from either occluded or minimally stenosed arteries?”
We sought evaluations of the performance and effectiveness of magnetic resonance angiography in carotid artery stenosis published between January 1990 and December 1999. This review was part of a larger one, in which we also sought articles discussing the use of magnetic resonance angiography in peripheral vessels. We searched the electronic databases Medline, Embase, HealthSTAR, Science Citation Index, Index to Scientific and Technical Proceedings, Cochrane Library, Inside (British Library), and Online Computer Library Centre by using the keywords magnetic resonance angiography and MRA (or accepted synonyms and abbreviations).7 We also conducted a hand search of 10 key journals in the fields of imaging and vascular disease. We examined the reference lists of all articles retrieved from the above sources.
We compiled a list of unique articles by eliminating duplicates identified from more than one database. We applied preliminary exclusion criteria within the bibliographic databases, and one reviewer then applied these and a second list of exclusion criteria (table 1). We retrieved the remaining articles, and two reviewers applied final inclusion criteria (table 2) independently, in a hierarchical manner from A to H. Inclusion criteria A-C and F identified articles that described robust primary research comparing the diagnostic performance of magnetic resonance angiography against conventional angiography (A) in terms of sensitivity and specificity (B) and at the specific thresholds of stenosis described in our introduction (C, F). Criterion D avoids results from the same study being included twice. The remaining criteria are indicators of the validity of the results. The results from articles that did not satisfy criterion E may have had verification bias; if the study group included asymptomatic participants (G) then patient cohort bias may have been present; and disease progression bias could result from a large time period between magnetic resonance angiography and the gold standard (H).8 Checklists covering study design, study group characteristics, technical details, and potential biases in executing the study were completed for each article satisfying the inclusion criteria A to D.
We included articles satisfying criteria A-D in the quantitative meta-analysis. Primary data (true positive, true negative, false positive, and false negative values) were extracted independently by two reviewers, and agreed by consensus, for the following clinical decision thresholds measured by using the NASCET criteria or similar4: 70-99% stenosed vessel (suitable for carotid endarterectomy) versus 0-69% stenosed or 100% occluded vessel (not suitable for carotid endarterectomy); 50-99% stenosed vessel (suitable for carotid endarterectomy) versus 0-49% stenosed or 100% occluded vessel (not suitable for carotid endarterectomy). Results derived with the very different ECST criteria were excluded from the meta-analysis.5
We plotted results from the independent studies on sensitivity versus 1−specificity axes to illustrate the scatter of results. We then combined the sensitivity and specificity results for independent studies into a summary receiver operating characteristic curve by using the method recommended by the Cochrane Screening and Diagnostic Tests Methods Working Group for meta-analysis of diagnostic test results.9–12 We used a logistic function to transform the sensitivity and specificity data. To prevent undefined values on logistic transformation, we applied a contingency correction of 0.01 to all true positive, false negative, true negative, and false positive values if any one was zero. We fitted a straight line to the transformed results by using an unweighted least squares fit, performed an inverse transformation on the fitted line, and plotted the result as a summary receiver operating characteristic curve.
The summary receiver operating characteristic curve is an excellent graphical summary, but for comparison purposes we calculated a further statistic—Q* and its 95% confidence interval.9 Q* is the point on the summary receiver operating characteristic curve where sensitivity and specificity have the same value; it represents the maximal joint sensitivity and specificity. Q* is a good summary value in this application as there is no particular disadvantage to sensitivity and specificity being equal: patients with false positive results needlessly undergo the risks of surgery, but patients with false negative results are denied the benefits of surgery. Q* would not be a good choice for describing a screening test, where the aim is to have no false negatives but a few false positives are acceptable.
We then performed a multiple linear regression analysis at the line fitting phase of the summary receiver operating characteristic analysis, to determine if any of five covariates had a significant effect, at the 95% level, on the fitted summary receiver operating characteristic curve.13 The five covariates were technique of magnetic resonance angiography, inclusion of articles that did not satisfy the inclusion criteria E-H, the risk of test or diagnostic review bias, the risk of verification bias, and the risk of withdrawal bias.14 As five variables were tested, the P value corresponding to 95% significance was 0.01.15 Statistical analyses were performed with SPSS for Windows (release 9.0.0).
We identified 16 185 articles with the initial broad search strategy. After we had removed duplicates, 7183 unique articles remained. The exclusion criteria reduced the number to 206 (table 1), and of 126 candidate articles on carotid artery stenosis 26 satisfied the inclusion criteria A to D.16–41 Only eight articles satisfied all the inclusion criteria A to H. 17 20 22 23 25 35 36 39 Six of these eight papers included results for the 70-99% threshold, 17 22 23 25 35 36 and two included results for the 50-99% threshold. 20 39
For the diagnosis of 70-99% stenosis (fig 1), four sets of results obtained by using contrast enhanced techniques were included, 26 32 34 36 together with 11 sets of results (from nine articles 19 23 25 27 29 34 35 37 41) obtained by using three dimensional time of flight and 10 sets of results16–38 obtained by using two dimensional time of flight techniques. Q* was 99% (95% confidence interval 98% to 100%). None of the variables tested in the multiple linear regression, including magnetic resonance angiography technique, was significant at the 95% level.
For the diagnosis of 50-99% stenosis (fig 2), no results obtained by using contrast enhanced techniques were included. Results from four studies using three dimensional time of flight techniques were included, 21 29 39 40 together with six sets of results (from five articles 20 24 28 31 33) obtained by using two dimensional time of flight. Q* was 90% (81% to 99%). None of the variables tested in the multiple linear regression was significant at the 95% level.
The patient populations of studies included in the quantitative meta-analysis were heterogeneous. Patient numbers ranged from 11 to 101 (mean=40). In all articles where sex distribution was reported most patients were men; in these articles the proportion of men ranged from 55% to 100% (mean=69%). Six articles did not report sex distribution. 17 22 26 29 30 35 The lower limit for age of patients ranged from 18 to 63 years (mean=43 years), and the upper limit ranged from 73 to 87 years (mean=80 years). Six articles did not report age range. 25 27 29 34 35 38 Eight articles stated that asymptomatic patients were included, 16 18 27–29 31 32 40 and 18 articles gave no information about symptoms. 16 17 19–26 28–30 33–37
Although many articles have been written about magnetic resonance angiography and carotid disease, little rigorous research has been conducted on the performance of magnetic resonance angiography in evaluating carotid artery stenosis. Small numbers of participants and inadequate details of study design mean that many studies included in this review have a potential for bias, but none of the factors tested in the multiple linear regression analysis had a significant effect on the results. Further sources of heterogeneity in patient populations (age, sex distribution, presenting symptoms) may influence patients' suitability for carotid endarterectomy, 5 42 and differences existed between the study groups included in the review, even among those that satisfied all the inclusion criteria.
Potential for bias in this review arose from the increasingly common practice of screening candidates for magnetic resonance angiography by using ultrasonography and proceeding only if this shows a stenosis greater than a predetermined value. Some of the results used in this review were probably obtained in groups of patients preselected in this way, but we were not able to determine this from the articles.
Ultimately, to show effectiveness, we need evidence of the impact of magnetic resonance angiography on clinical decision making and on outcomes in patients. 43 44 Comparative studies, with patients randomised to magnetic resonance angiography or conventional angiography, could be used to gather evidence of the impact on decision making. In this review, however, we found no studies that compared magnetic resonance angiography with conventional angiography for surgical decision making or outcomes. Although high quality research is needed, especially full cost effectiveness studies and evaluation of new magnetic resonance angiography techniques, further large trials involving conventional angiography are unlikely to be undertaken. Modelling, using existing evidence, may be the way forward.
Degree of stenosis
To be able to determine whether a patient is a suitable candidate for carotid endarterectomy, a diagnostic test must distinguish severely (>70%) stenosed or moderately (>50%) stenosed arteries (which are suitable for carotid endarterectomy) from both minimally stenosed (0-69% or 0-49%) and occluded arteries (100%), which are not suitable for carotid endarterectomy. 4 5 Candidate articles often failed to assess magnetic resonance angiography in these terms. We excluded 20 articles from the review because they did not classify operable carotid artery stenosis as 70-99% or 50-99%. Of these, seven articles also failed to report the diagnostic performance of magnetic resonance angiography in separating severely stenosed from occluded arteries. This can be an important distinction in selection for surgery, especially if ultrasonography has indicated a small amount of flow.
Some authors have recommended that the magnitude of the likely benefit of carotid endarterectomy in cases of moderate to severe stenosis should be assessed by using more restricted bands of stenosis.45 Future trials with restricted bands might show which stenosis thresholds, as determined by magnetic resonance angiography, are associated with surgical benefit, in addition to determining the reliability of magnetic resonance angiography in detecting patients suitable for surgery. This research might be an ideal testing ground for tracker studies that evaluate fast changing technologies.46
Our review does not support the use of magnetic resonance angiography to select surgical candidates with 50-99% stenosis. The 95% confidence interval for Q* extended from 81% to 99%, and only two of the articles whose results were included in the meta-analysis satisfied the inclusion criteria related to validity. It would be advisable for users of magnetic resonance angiography to ensure that rigorous training and audit are in place, including feedback from surgeons and continuing quality control comparisons with ultrasonography.
Our results indicate that magnetic resonance angiography is very effective for detecting 70-99% stenosis as defined by conventional angiography. Q* was 99% (98% to 100%). Although there is a promising trend towards better performance from contrast enhanced methods, further research is essential as only four articles were included in this review and no significant difference was found between the results obtained by using the three main techniques.
We thank all members of the review team for their contributions to the conduct of the systematic review, to hand searching, and to translation.
Contributors: MEW wrote the first draft of the paper and, with SK, performed the literature search and data extraction and participated in designing the review and writing the paper. EB coordinated the review and the writing of the paper and performed the data analysis. She will be guarantor. JMB, MJG, CMA, JFMM, LMD, and JC participated in the design and execution of the review and in writing the paper. MAS participated in writing the paper.
Funding Financial support from the secretary of state for health under the NHS Health Technology Assessment Programme (97/13/04). The views and opinions expressed do not necessarily reflect those of the secretary of state for health. Leeds Teaching Hospitals NHS Trust received funding from the NHS Executive; the views expressed in this publication are those of the authors and not necessarily those of the NHS Executive.
Competing interests JFMM has been reimbursed for presenting material to meetings organised by the pharmaceutical industry (Schering); JFMM has received funds for research on magnetic resonance angiography from both Philips Medical Systems and Schering UK; JFMM has received funding from Philips Medical Systems for a part time research assistant; MAS has collaborative links with Philips Medical Systems, who have provided research support for magnetic resonance angiography.