Original ArticleA systematic review finds that diagnostic reviews fail to incorporate quality despite available tools
Introduction
There is a growing interest in the systematic review and quantitative synthesis of research evaluating diagnostic tests. A crucial step in the process of reviewing is the critical appraisal of the quality of studies [1], [2], [3], [4]. Study quality can be used as a criterion for inclusion, or as a potential source of heterogeneity in study results. Study quality, however, has proven to be an elusive concept. Generally speaking, study quality can involve one or more of four aspects. The first is the potential for bias. What is the fit between the study design and the study purpose? Shortcomings in the design and conduct of diagnostic studies can lead to exaggerated estimates of diagnostic accuracy. Evidence is accumulating on the amount and direction of these potential threats to validity [5], [6]. The second aspect of quality is the conduct of the study. How well was the study planned, and what are the discrepancies between what was planned and what was actually performed? A third aspect of quality is the applicability of the results. Are the results of the study applicable to the clinical problem that the reader is interested in? A fourth aspect is the quality of reporting. Does the report or article contain the information needed to assess the potential for bias, the quality of conduct of the study, and the applicability of the results? Some people may see quality as involving all four aspects; others may see it as concerned with only one or two.
Several tools for evaluating the quality of diagnostic studies, some quite different from one another, have been presented in the literature. We report the results of a systematic review of these existing tools to evaluate the quality of diagnostic test evaluations. In a second systematic review, we have documented the ways in which quality has been incorporated in published systematic review of diagnostic tests.
Section snippets
Materials and methods
For the first systematic review, electronic databases including Medline, Embase, Biosis, and the methodological databases of both the Centre for Reviews and Dissemination (CRD) and the Cochrane Collaboration were systematically searched from database inception to April 2001 to identify existing quality assessment tools. Full details of the search strategy are provided elsewhere [7]. Methodological experts in the area of diagnostic tests were contacted for additional studies. No language
Search results
A total of 91 quality-related tools were identified, of which 67 could be classified as tools designed to assess the quality of primary studies of diagnostic accuracy. Of these, 58 were described only within the methods section of a particular systematic review [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47],
Discussion
These two systematic reviews highlight the large variation in tools currently used to assess the quality of studies of diagnostic accuracy, and the different ways in which systematic reviews have incorporated quality.
We were able to identify 91 quality assessment tools. Only two of these provided details on how they had been developed, and none of the tools appear to have been systematically evaluated. There is an obvious lack of agreement about the term quality and why this should be measured.
Conclusion
Existing tools used to assess the quality of diagnostic accuracy studies provide limited detail on the method for their development, and on the aspects of quality that the tools aimed to address. None of these tools appears to have been systematically evaluated. The wide variation in the existing tools to assess quality of diagnostic studies, and the various ways in which study quality differences have been taken into account in systematic reviews, show the need for guidance in this area.
Acknowledgments
Our thanks go to Kath Wright (CRD) for conducting literature searches. This review was commissioned and funded by the NHS R&D Health Technology Assessment Programme. The views expressed in this review are those of the authors and not necessarily those of the Standing Group, the Commissioning Group, or the Department of Health.
References (116)
- et al.
Summary receiver operating characteristic curves as a technique for meta-analysis of the diagnostic performance of duplex ultrasonography in peripheral arterial disease
Acad Radiol
(1996) - et al.
Methodologic standards for diagnostic test research in pulmonary medicine
Chest
(1998) - et al.
The accuracy of serum chlamydial antibodies in the diagnosis of tubal pathology: a meta-analysis
Fertil Steril
(1997) - et al.
The accuracy of hysterosalpingography in the diagnosis of tubal pathology: a meta-analysis
Fertil Steril
(1995) - et al.
Diagnostic value of tests that discriminate between exudative and transudative pleural effusions
Chest
(1997) - et al.
The performance of CA-125 measurement in the detection of endometriosis: a meta-analysis
Fertil Steril
(1998) - et al.
Accuracy of MR imagining in the work-up of suspicious breast lesions: a diagnostic meta-analysis
Acad Radiol
(1999) - et al.
Clinical tests of the sacroiliac joint: a systematic methodological review. Part 1: Reliability
Man Ther
(2000) - et al.
Methodologic problems of exercise testing for coronary artery disease: groups, analysis and bias
Am J Cardiol
(1980) - et al.
Systematic reviews to evaluate diagnostic tests
Eur J Obstet Gynecol Reprod Biol
(2001)