Intended for healthcare professionals

CCBYNC Open access
Research Methods & Reporting

Anticipating missing reference standard data when planning diagnostic accuracy studies

BMJ 2016; 352 doi: (Published 09 February 2016) Cite this as: BMJ 2016;352:i402
  1. Christiana A Naaktgeboren, assistant professor1,
  2. Joris A H de Groot, assistant professor1,
  3. Anne W S Rutjes, senior clinical epidemiologist2 3,
  4. Patrick M M Bossuyt, professor4,
  5. Johannes B Reitsma, associate professor1,
  6. Karel G M Moons, professor1
  1. 1Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, 3584 CG Utrecht, Netherlands
  2. 2CTU Bern, Department of Clinical Research, University of Bern, Switzerland
  3. 3Institute of Social and Preventive Medicine, University of Bern, Switzerland
  4. 4Department of Clinical Epidemiology and Biostatistics, Academic Medical Center, University of Amsterdam, Amsterdam, Netherlands
  1. Correspondence to: J B Reitsma j.b.reitsma-2{at}
  • Accepted 30 December 2015

Results obtained using a reference standard may be missing for some participants in diagnostic accuracy studies. This paper looks at methods for dealing with such missing data when designing or conducting a prospective diagnostic accuracy study

Summary points

  • Missing reference standard results—that is, missing data on the target disease status—are common in diagnostic accuracy studies

  • Analyses that include only the study participants for whom the target disease status is actually measured may produce biased estimates of accuracy

  • Several statistical methods to reduce this bias are available; however, they all rely on assumptions about the pattern of missing outcomes, which are sometimes unverifiable

  • This paper provides an overview of the different patterns of missing data on the reference standard, the recommended corresponding solutions, and the specific measures that can be taken before and during a prospective diagnostic study to enhance the validity and interpretation of these solutions

The problem: missing reference standard data

Diagnostic studies typically evaluate the accuracy of one or more tests, markers, or models by comparing their results with those of, ideally, a “gold” reference test or standard.1 2 In such studies, the outcome—that is, the presence or absence of the target disease as determined by the chosen reference standard—is often missing in some of the study participants. This is known as partial verification.3 4 When only the participants who received the reference standard are included in the analysis (complete case analysis), estimates of the accuracy of the diagnostic test(s), marker(s), or model(s) under study, such as the sensitivity, specificity, predictive values, likelihood ratios, or C index, can be biased.5 6 7 8

There are many reasons why missing reference standard results may occur in diagnostic studies, as well as various approaches to deal with these missing outcomes in the statistical analysis.3 4 8 9 10 11 12 13 14 15 16 …

View Full Text