Original Article
Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews

https://doi.org/10.1016/j.jclinepi.2005.02.022Get rights and content

Abstract

Background and Objectives

Studies of diagnostic accuracy most often report pairs of sensitivity and specificity. We demonstrate the advantage of using bivariate meta-regression models to analyze such data.

Methods

We discuss the methodology of both the summary Receiver Operating Characteristic (sROC) and the bivariate approach by reanalyzing the data of a published meta-analysis.

Results

The sROC approach is the standard method for meta-analyzing diagnostic studies reporting pairs of sensitivity and specificity. This method uses the diagnostic odds ratio as the main outcome measure, which removes the effect of a possible threshold but at the same time loses relevant clinical information about test performance. The bivariate approach preserves the two-dimensional nature of the original data. Pairs of sensitivity and specificity are jointly analyzed, incorporating any correlation that might exist between these two measures using a random effects approach. Explanatory variables can be added to the bivariate model and lead to separate effects on sensitivity and specificity, rather than a net effect on the odds ratio scale as in the sROC approach. The statistical properties of the bivariate model are sound and flexible.

Conclusion

The bivariate model can be seen as an improvement and extension of the traditional sROC approach.

Introduction

Diagnostic accuracy studies are a vital step in the evaluation of diagnostic technologies [1], [2], [3] Accuracy studies measure the level of agreement between the results of a test under evaluation and that of the reference standard. There are several different measures of diagnostic accuracy [4], [5], but the majority of diagnostic accuracy studies present estimates of sensitivity and specificity, either alone or in combination with other measures [6].

Because the majority of diagnostic papers report estimates of sensitivity and specificity, meta-analytic approaches have focused on these measures [6], [7], [8], [9], [10], [11], [12], [13]. Pooling pairs of sensitivity and specificity is not straightforward, because these measures are often negatively correlated within studies.

The summary Receiver Operating Characteristic (sROC) approach has become the method of choice for the meta-analysis of studies reporting pairs of sensitivity and specificity [9], [12], [14], [15], [16], [17], [18]. The sROC approach converts each pair of sensitivity and specificity into a single measure of accuracy, the diagnostic odds ratio [19]. The disadvantage of a single measure of diagnostic accuracy is that it does not distinguish between the ability of detecting the sick (sensitivity) and identifying the well (specificity). Discriminating between these abilities is important to determine the optimal use of a test in clinical practice. The bivariate model we propose has the distinct advantage of preserving the two-dimensional nature of the underlying data. It can also produce summary estimates of sensitivity and specificity, acknowledging any possible (negative) correlation between these two measures. We will discuss both approaches and illustrate their use by reanalyzing the data from a published meta-analysis [20].

Section snippets

Pooling pairs of sensitivity and specificity: why simple methods fail

Diagnostic reviews start with a set of individual studies presenting estimates of sensitivity and specificity. One intuitive approach is to do separate pooling of sensitivity and specificity using standard methods for proportions. However, sensitivity and specificity are often negatively correlated within studies, and ignoring this correlation would be inappropriate [7], [11], [12].

One possible cause for this negative correlation between sensitivity and specificity is that studies may have used

Summary ROC approach

We provide a short description of the sROC approach as outlined by Moses and Littenberg. More details can be found elsewhere [9], [12], [14], [15], [16], [17], [18].

The sROC approach starts with plotting the observed pairs of sensitivity and specificity of each study in ROC space (see Fig. 1). The aim of the sROC approach is to find a smooth curve through these points. The key step is to transform the TPR (sensitivity) and FPR (1 − specificity) scale of the ROC graph so that the relation

Bivariate model

The bivariate model uses a different starting point for the meta-analysis of pairs of sensitivity and specificity. Rather than transforming these two distinct outcome measures into a single indicator of diagnostic accuracy as in the sROC approach, the bivariate model preserves the two-dimensional nature of the data throughout the analysis.

The bivariate model is based on the following line of reasoning [10], [24], [25]. We assume that the sensitivities from individual studies (after logit

General discussion of methods of meta-analysis

In this section we discuss the statistical methods for meta-analysis of studies of diagnostic accuracy by comparing them on a few key items. These are: (1) the choice of outcome measure; (2) the effect of covariates; (3) the statistical properties of the model.

References (46)

  • J.D.F. Habbema et al.

    Analysis of data on the accuracy of diagnostic tests

  • H. Honest et al.

    Reporting of measures of accuracy in systematic reviews of diagnostic literature

    BMC Health Serv Res

    (2002)
  • S.D. Walter et al.

    Meta-analysis of screening data: a survey of the literature

    Stat Med

    (1999)
  • J.G. Lijmer et al.

    Empirical evidence of design-related bias in studies of diagnostic tests

    JAMA

    (1999)
  • J.W. Kardaun et al.

    Comparative diagnostic performance of three radiological procedures for the detection of lumbar disk herniation

    Methods Inf Med

    (1990)
  • C.M. Rutter et al.

    A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations

    Stat Med

    (2001)
  • L.E. Moses et al.

    Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations

    Stat Med

    (1993)
  • J.J. Deeks

    Systematic reviews in health care: systematic reviews of evaluations of diagnostic and screening tests

    BMJ

    (2001)
  • V. Hasselblad et al.

    Meta-analysis of screening and diagnostic tests

    Psychol Bull

    (1995)
  • L. Irwig et al.

    Guidelines for meta-analyses evaluating diagnostic tests

    Ann Intern Med

    (1994)
  • B. Littenberg et al.

    Estimating diagnostic accuracy from multiple conflicting reports: a new meta-analytic method

    Med Decis Making

    (1993)
  • A.S. Midgette et al.

    A meta-analytic method for summarizing diagnostic test performances: receiver-operating- characteristic-summary point estimates

    Med Decis Making

    (1993)
  • S.D. Walter

    Properties of the summary receiver operating characteristic (SROC) curve for diagnostic test data

    Stat Med

    (2002)
  • Cited by (2502)

    View all citing articles on Scopus
    View full text