Original Article
Single data extraction generated more errors than double data extraction in systematic reviews

https://doi.org/10.1016/j.jclinepi.2005.11.010Get rights and content

Abstract

Background and Objective

To conduct a pilot study to compare the frequency of errors that accompany single vs. double data extraction, compare the estimate of treatment effect derived from these methods, and compare the time requirements for these methods.

Methods

Reviewers were randomized to the role of data extractor or data verifier, and were blind to the study hypothesis. The frequency of errors associated with each method of data extraction was compared using the McNemar test. The data set for each method was used to calculate an efficacy estimate by each method, using standard meta-analytic techniques. The time requirement for each method was compared using a paired t-test.

Results

Single data extraction resulted in more errors than double data extraction (relative difference: 21.7%, P = .019). There was no substantial difference between methods in effect estimates for most outcomes. The average time spent for single data extraction was less than the average time for double data extraction (relative difference: 36.1%, P = .003).

Conclusion

In the case that single data extraction is used in systematic reviews, reviewers and readers need to be mindful of the possibility for more errors and the potential impact these errors may have on effect estimates.

Introduction

The purpose of a systematic review is to provide a comprehensive, valid, and reliable synthesis of the evidence regarding a particular question. There are numerous measures used during the conduct of systematic reviews to ensure that they meet these criteria. The stage of data extraction involves transcribing information reported in the primary studies to a standard form that has been developed to capture all relevant information in a format specific to the review question. This stage is critical to the results and conclusions of the review, as the subsequent analysis and interpretation stem from the information extracted.

Recent investigations have raised questions regarding the accuracy and reliability of the data extraction process in systematic reviews. Strang et al. [1] found substantial discordance between two experts extracting data from 12 clinical trials. Discrepancies occurred with respect to extraction of samples sizes, design, start and end dates, selection criteria, and secondary outcomes. The authors recommended enhanced rigor in data extraction including systematic, double extraction. In another investigation, data extraction was repeated by a medical statistician for 28 systematic reviews published by the Cochrane Cystic Fibrosis and Genetic Disorders Group [2]. The rate of inaccuracies was found to be “unacceptably high,” which has led to new policies to reduce errors associated with data extraction. Contrary to these findings, Haywood et al. [3] reported good agreement in data extraction between three reviewers with different expertise using an electronic database with careful instruction and training.

The currently recommended method of data extraction in systematic reviews is for two researchers to independently extract data and come to consensus over discrepancies through discussion or in consultation with a third party (double data extraction) [4], [5]. The consensus process involves resolving differences between two sets of data extraction. This method is labor-intensive, time-consuming, and costly. An alternative method is for one researcher to independently extract data, a second researcher to verify the data extraction, and consensus over discrepancies is reached through discussion or in consultation with a third party (single data extraction).

To our best knowledge, there is no empirical evidence regarding the types and magnitude of errors arising from these two methods and their relative impact on the results of meta-analysis. Our objective was to conduct a pilot study to describe and compare the types and frequency of errors that accompany single vs. double data extraction; compare the estimate of treatment effect derived from single vs. double data extraction; and compare the time required for single vs. double data extraction.

Section snippets

Methods

Our Center recently conducted a systematic review of the efficacy and safety of melatonin for the management of sleep disorders [6], under contract with the Agency for Healthcare Research and Quality. The efficacy review included 30 randomized controlled trials of the effect of melatonin on people with sleep disorders. The outcomes of interest were sleep onset latency, sleep efficiency, wakefulness after sleep onset, total sleep time, sleep quality, REM latency and percent time in REM sleep.

Rate of disagreement

The overall rate of disagreement between the two methods was 28.0 % [95% confidence interval (CI): 25.4, 30.7] (Table 1). The rate ranged from 11.1 to 47.2% among studies (median: 27.8%) and from 0.0 to 76.7% among variables (median: 23.3%). The variables with the lowest rates of disagreement were language of publication, gender distribution of population, type of sleep restriction disorder, and study interventions (0.0%). The variable with the highest rate of disagreement was outcome reporting

Discussion

The rate of disagreement between methods of data extraction was high. A number of factors may have contributed to this finding. The majority of reviewers had minimal knowledge of the content area, and some reviewers had minimal experience in data extraction and data verification, which may have led to an increase in the number of errors, and therefore, in the number of disagreements. Although detailed instruction was provided to reviewers with respect to the items to be extracted for each

Conclusion

In summary, single data extraction generates more errors but requires less time than double data extraction. The decision to employ single vs. double data extraction for a review depends on a number of factors, including the time and human resources available for the review. If there are adequate resources to employ double data extraction, then this method should be used. In the case that single data extraction is used, reviewers and readers need to be mindful of the possibility for more errors

Acknowledgments

N.B. participated in designing the study, data collection, data analysis, and writing and editing the manuscript. L.H. participated in designing the study, data analysis, and writing and editing the manuscript. B.V. participated in designing the study, statistical analysis of data, and writing and editing the manuscript. L.T. participated in writing and editing the manuscript. T.P.K. reviewed and provided feedback on the manuscript. The authors acknowledge Nicola Hooton, Brad Johnston, Daniel

References (6)

  • Strang WN, Boissel P, Uberla K. Inter-reader variation. 2nd International Conference. Scientific Basis of Health...
  • Jones A. Improving the reliability of data extraction [abstract]. XI Cochrane Colloquium: evidence, health care and...
  • K.L. Haywood et al.

    Reviewing measures of outcome: reliability of data extraction

    J Eval Clin Pract

    (2004)
There are more references available in the full text version of this article.

Cited by (0)

View full text