Brief Report
High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews

https://doi.org/10.1016/j.jclinepi.2004.11.024Get rights and content

Abstract

Background and Objective

Extracting data from primary articles is an essential component in conducting systematic reviews. Incorrect data extraction can lead to false conclusions. The objective of this study was to retrospectively repeat the data extraction in all systematic reviews conducted by the Cochrane Cystic Fibrosis and Genetic Disorders Group.

Study Design and Setting

For each review, data extraction was conducted, by an experienced statistician, for the same publications used by the reviewers. Results were compared with those obtained by the reviewers.

Results

Errors were found in 20 of 34 reviews, including incorrect calculations made when converting data in primary articles into data required for the review (2 reviews) and misinterpretation of data that were reported in the primary article (7 reviews). All data-handling errors led to changes in the summary results, but none of these affected the review conclusions.

Conclusions

Important errors were identified in a high proportion of reviews. A variety of problems relating to the reporting of results within a review were identified, but these did not lead to substantial changes in any conclusion.

Introduction

Systematic review methodology has been developed to minimize biases and random errors, using rigorous and explicit measures [1]. Generally, the quality of Cochrane systematic reviews is good [2]. Even in a well-written review, however, it is difficult to assess whether errors in data handling or reporting have occurred.

Many Cochrane systematic reviews are undertaken by reviewers with limited or no research training or experience in statistical analysis. The editorial process does not generally allow scrutiny of data extraction by, for example, referring back to the original trial reports.

Our objective was to assess the quality of data extraction and reporting in 42 reviews from one Cochrane Collaborative Review Group as published in Issue 4, 2003 of The Cochrane Library [3].

Section snippets

Methods

The editorial process for the Cochrane Cystic Fibrosis and Genetic Disorders Group is as follows. When a title is registered, a review is allocated a contact editor, who, alongside the Review Group Coordinator, oversees the production of the review. To reduce the likelihood of error, it is recommended that assessment of study eligibility, methodological quality, and data extraction be carried out independently by at least two reviewers [4].

At least four external referees comment on each

Results

Eight of the 42 reviews included no studies. Table 1 summarizes data-handling and reporting errors in the remaining 34 reviews. Data extraction was undertaken independently by at least two reviewers in 30 reviews; by one reviewer in two; and was not stated in two.

Errors were found in 20 reviews. Within two reviews, incorrect calculations were made converting data in primary articles into data required for the review (Table 1, a to b). Misinterpretation of data reported in the primary article

Conclusion

We have shown that when an experienced statistician assessed whether data were extracted and reported correctly, errors were identified in a high proportion of reviews (20 out of 34), but these did not lead to substantial changes in any conclusion.

Correct data extraction is fundamental to the integrity of meta-analysis but, to our knowledge, has never been formally assessed in Cochrane systematic reviews. Data-handling errors (Table 1, a to n) would not usually be identified by referees, and

Acknowledgments

The present study was conducted under the auspices of the Department of Child Health and the Centre of Medical Statistics and Health Evaluation, University of Liverpool, and the Wolfson Institute of Preventive Medicine, University of London.

References (5)

  • O. Olsen et al.

    Quality of Cochrane reviews: assessment of sample from 1998

    BMJ

    (2001)
There are more references available in the full text version of this article.

Cited by (80)

  • Methodological standards for conducting and reporting meta-analyses: Ensuring the replicability of meta-analyses of pharmacist-led medication review

    2022, Research in Social and Administrative Pharmacy
    Citation Excerpt :

    Jones et al. performed data extraction of 34 meta-analyses conducted by the Cochrane Cystic Fibrosis and Genetic Disorders Group and found errors in 20 of them (58.8%), which included incorrect calculations when converting data in primary articles into data required for the review and misinterpretation of data that were reported in the primary article. However, as in the present study, although these errors led to changes in the summary results, they did not affect the final conclusions, probably given the large uncertainty of effect sizes.47 Ford et al. assessed eight systematic reviews and meta-analyses of pharmaceutical interventions for irritable bowel syndrome and found errors in both applications of eligibility criteria and dichotomous data extraction in all of them.

  • Best practices when conducting and reporting a meta-analysis

    2022, Contemporary Research Methods in Pharmacy and Health Services
View all citing articles on Scopus
View full text