Review Article
A systematic review identifies a lack of standardization in methods for handling missing variance data

https://doi.org/10.1016/j.jclinepi.2005.08.017Get rights and content

Abstract

Background and Objectives

To describe and critically appraise available methods for handling missing variance data in meta-analysis (MA).

Methods

Systematic review. MEDLINE, EMBASE, Web of Science, MathSciNet, Current Index to Statistics, BMJ SearchAll, The Cochrane Library and Cochrance Colloquium proceedings, MA texts and references were searched. Any form of text was included: MA, method chapter, or otherwise. Descriptions of how to implement each method, the theoretic basis and/or ad hoc motivation(s), and the input and output variable(s) were extracted and assessed. Methods may be: true imputations, methods that obviate the need for a standard deviation (SD), or methods that recalculate the SD.

Results

Eight classes of methods were identified: algebraic recalculations, approximate algebraic recalculations, imputed study-level SDs, imputed study-level SDs from nonparametric summaries, imputed study-level correlations (e.g., for change-from-baseline SD), imputed MA-level effect sizes, MA-level tests, and no-impute methods.

Conclusion

This work aggregates the ideas of many investigators. The abundance of methods suggests a lack of consistency within the systematic review community. Appropriate use of methods is sometimes suspect; consulting a statistician, early in the review process, is recommended. Further work is required to optimize method choice to alleviate any potential for bias and improve accuracy. Improved reporting is also encouraged.

Introduction

Meta-analysis is an efficient way of analytically combining the results of individual studies together to provide an overall estimate of an intervention's effectiveness along with a measure of its precision. Selective reporting, present in many forms (e.g., omittance of publications, outcomes, or subgroups), will decrease statistical power and may distort the magnitude and possibly the direction of meta-analysed results. In a recent study that compared Danish protocols of randomized controlled trials to their published manuscripts, Chan et al. [1] found incompletely reported results for 92% of efficacy outcomes and 81% of harm outcomes. Similar results have been reported for other jurisdictions [2]. Incomplete reporting of continuous outcomes is often manifested in the omission of standard deviations (SDs). Streiner and Joffe [3] found that only 9 out of 69 randomized controlled trials in their systematic review (SR) of antidepressants reported standard deviations; Song et al. [4] found 20 of 33 studies reported SDs in their SR of selective serotonin reuptake inhibitors for depression.

To the best of our knowledge, a comprehensive review on the effects of omitting studies that did not report SDs has not been reported. Regardless, omitting studies that fail to report SDs is wasteful. SDs are used in the calculation of the precision of the overall pooled estimate; thus, missing SDs reduce the power of the meta-analysis (MA). Additionally, SDs are used in the weighted calculation of the overall pooled estimate. Omitting studies due to systematically missing SDs may induce bias in this estimate. In primary studies (e.g., surveys, randomized controlled trials), Schafer [5] suggests that ignoring a case loss of 5% or less because of missing data is a reasonable solution to the missing data problem. In secondary studies (i.e., meta-analysis), where the number of included studies tends to be small, any omitted study may amount to a significant loss.

Meta-analysts are forced to improvise and/or use other summary measures to derive an SD estimate to include studies with missing SDs. When SDs cannot be algebraically recalculated from reported data, meta-analysts have suggested and used a myriad of methods to impute SD (fill in SDs with plausible values) to attenuate any loss in power and to avoid bias. However, many of these methods have not been theoretically derived or empirically tested. It has not been known how many such methods exist, whether different methods provide different variance imputation “corrections,” and whether different imputation methods influence the overall conclusions of a meta-analysis.

Section snippets

Objective

To describe, and critically appraise, methods for handling missing variance data in meta-analysis.

Search strategy

We developed a comprehensive search strategy to identify all relevant documents regardless of publication status. Combinations of the following sets of search terms were used: variance, standard deviation, standard error and imput, miss, deriv. We searched the following electronic databases: MEDLINE (1966 to May 17, 2002), EMBASE (1988 to May 17, 2002), Web of Science (May 3, 2002), MathSciNet (May 17, 2002), and Current Index to Statistics (September 17, 2002). We performed full-text

Results

The search yielded 822 documents (Fig. 1), 6% from references of included documents. One hundred fifty-three reports were included: 73% were SRs, and 27% were from method papers or textbooks (complete listing of citations are available from N.W.). Documents were excluded if they were duplicate papers, not retrievable, or did not contain a relevant method.

We classified the methods into eight groups: algebraic recalculation, approximate algebraic recalculation, study-level imputation, study-level

Discussion

This work aggregates the ideas of many researchers. The abundance of methods suggests a lack of communication across the systematic review community. At least two matters should be considered when selecting a method of imputation: (1) what information is already known about the missing SDs, and (2) the mechanism that caused the data to be missing. Harrell [52], among others, discuss three mechanisms for missing data. Missing completely at random (MCAR) suggests that the missing data is

Limitations

Our prevalence statistics (Table 1) are limited by the simple search strategy we used in the Cochrane Library and BMJ SearchAll; a hand search of an entire issue of the Cochrane Library and a highly sensitive search strategy to find systematic reviews in BMJ SearchAll may prove more productive. Also, our findings are limited by how comprehensively the authors (given the constraints imposed by journals) reported their strategy for handling missing SD. In addition, the psychologic literature has

Acknowledgments

We were supported through an operating grant (MOP57880) from the Canadian Institutes of Health Research. We thank Ellen Crumley and Margaret Sampson for assistance with literature searching. We also thank Jennifer Houseman, John Russell, and Philip Berry for their assistance with text retrieval and citation management.

References (53)

  • G.V. Glass et al.

    Measuring study findings

  • J. Deeks

    Are you sure that's a standard deviation? Part I & II

    Cochrane News

    (1998)
  • S. Martins et al.

    Iron therapy for improving psychomotor development and cognitive function in children under the age of three with iron deficiency anaemia (Cochrane Review)

  • S. Greenland

    Quantitative methods in the review of epidemiologic literature

    Epidemiol Rev

    (1987)
  • D. Allison et al.

    Antipsychotic-induced weight gain: a comprehensive research systhesis

    Am J Psychiatry

    (1999)
  • L. Brion et al.

    Intravenous or enteral loop diuretics for preterm infants with (or developing) chronic lung disease (Cochrane Review)

  • S.F. Buck

    A method of estimation of missing values in multivariate data suitable for use with an electronic computer

    J R Stat Soc

    (1960)
  • M.B. Bracken

    Statistical methods for analysis of effects of treatment in overviews of randomized trials

  • C.S. Spooner et al.

    Nedocromil sodium for preventing exercise-induced bronchoconstriction (Cochrane Review)

  • E.S. Pearson

    The application of statistical methods to industrial standardization and quality control

    (1960)
  • A. Vickers et al.

    Massage for promoting growth and development of preterm and/or low birth weight infants (Cochrane Review)

  • M. Suarez-Almazor et al.

    Penicillamine for treating rheumatoid arthritis (Cochrane Review)

  • D.G. Altman et al.

    Detecting skewness from summary information

    Br Med J

    (1996)
  • T.D. Pigott

    Methods for handling missing data in research synthesis

  • A.J. Sutton et al.

    Missing data

  • A. Cranney et al.

    Meta-analyses of therapies for postmenopausal osteoporosis. I. Systematic reviews of randomized trials in osteoporosis: Introduction and methodology

    Endocr Rev

    (2002)
  • Cited by (222)

    View all citing articles on Scopus
    View full text