Review ArticleA systematic review identifies a lack of standardization in methods for handling missing variance data
Introduction
Meta-analysis is an efficient way of analytically combining the results of individual studies together to provide an overall estimate of an intervention's effectiveness along with a measure of its precision. Selective reporting, present in many forms (e.g., omittance of publications, outcomes, or subgroups), will decrease statistical power and may distort the magnitude and possibly the direction of meta-analysed results. In a recent study that compared Danish protocols of randomized controlled trials to their published manuscripts, Chan et al. [1] found incompletely reported results for 92% of efficacy outcomes and 81% of harm outcomes. Similar results have been reported for other jurisdictions [2]. Incomplete reporting of continuous outcomes is often manifested in the omission of standard deviations (SDs). Streiner and Joffe [3] found that only 9 out of 69 randomized controlled trials in their systematic review (SR) of antidepressants reported standard deviations; Song et al. [4] found 20 of 33 studies reported SDs in their SR of selective serotonin reuptake inhibitors for depression.
To the best of our knowledge, a comprehensive review on the effects of omitting studies that did not report SDs has not been reported. Regardless, omitting studies that fail to report SDs is wasteful. SDs are used in the calculation of the precision of the overall pooled estimate; thus, missing SDs reduce the power of the meta-analysis (MA). Additionally, SDs are used in the weighted calculation of the overall pooled estimate. Omitting studies due to systematically missing SDs may induce bias in this estimate. In primary studies (e.g., surveys, randomized controlled trials), Schafer [5] suggests that ignoring a case loss of 5% or less because of missing data is a reasonable solution to the missing data problem. In secondary studies (i.e., meta-analysis), where the number of included studies tends to be small, any omitted study may amount to a significant loss.
Meta-analysts are forced to improvise and/or use other summary measures to derive an SD estimate to include studies with missing SDs. When SDs cannot be algebraically recalculated from reported data, meta-analysts have suggested and used a myriad of methods to impute SD (fill in SDs with plausible values) to attenuate any loss in power and to avoid bias. However, many of these methods have not been theoretically derived or empirically tested. It has not been known how many such methods exist, whether different methods provide different variance imputation “corrections,” and whether different imputation methods influence the overall conclusions of a meta-analysis.
Section snippets
Objective
To describe, and critically appraise, methods for handling missing variance data in meta-analysis.
Search strategy
We developed a comprehensive search strategy to identify all relevant documents regardless of publication status. Combinations of the following sets of search terms were used: variance, standard deviation, standard error and imput∗, miss∗, deriv∗. We searched the following electronic databases: MEDLINE (1966 to May 17, 2002), EMBASE (1988 to May 17, 2002), Web of Science (May 3, 2002), MathSciNet (May 17, 2002), and Current Index to Statistics (September 17, 2002). We performed full-text
Results
The search yielded 822 documents (Fig. 1), 6% from references of included documents. One hundred fifty-three reports were included: 73% were SRs, and 27% were from method papers or textbooks (complete listing of citations are available from N.W.). Documents were excluded if they were duplicate papers, not retrievable, or did not contain a relevant method.
We classified the methods into eight groups: algebraic recalculation, approximate algebraic recalculation, study-level imputation, study-level
Discussion
This work aggregates the ideas of many researchers. The abundance of methods suggests a lack of communication across the systematic review community. At least two matters should be considered when selecting a method of imputation: (1) what information is already known about the missing SDs, and (2) the mechanism that caused the data to be missing. Harrell [52], among others, discuss three mechanisms for missing data. Missing completely at random (MCAR) suggests that the missing data is
Limitations
Our prevalence statistics (Table 1) are limited by the simple search strategy we used in the Cochrane Library and BMJ SearchAll; a hand search of an entire issue of the Cochrane Library and a highly sensitive search strategy to find systematic reviews in BMJ SearchAll may prove more productive. Also, our findings are limited by how comprehensively the authors (given the constraints imposed by journals) reported their strategy for handling missing SD. In addition, the psychologic literature has
Acknowledgments
We were supported through an operating grant (MOP57880) from the Canadian Institutes of Health Research. We thank Ellen Crumley and Margaret Sampson for assistance with literature searching. We also thank Jennifer Houseman, John Russell, and Philip Berry for their assistance with text retrieval and citation management.
References (53)
- et al.
Variance imputation for overviews of clinical trials with continuous response
J Clin Epidemiol
(1992) - et al.
Cholesterol-lowering effects of dietary fiber: a meta-analysis
Am J Clin Nutr
(1999) - et al.
A meta analysis of the effects of dietary protein restriction on the rate of decline in renal function
Am J Kidney Dis
(1998) - et al.
Beyond classical meta-analysis: can inadequately reported studies be included?
Drug Discov Today
(2004) - et al.
Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles
JAMA
(2004) - et al.
Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
Can Med Assoc J
(2004) - et al.
The adequacy of reporting randomized, controlled trials in the evaluation of antidepressants
Can J Psychiatry
(1998) - et al.
Selective serotonin reuptake inhibitors: meta-analysis of efficacy and acceptability
Br Med J
(1993) Analysis of incomplete multivariate data
(1997)- et al.
Practical meta-analysis
(2001)
Measuring study findings
Are you sure that's a standard deviation? Part I & II
Cochrane News
Iron therapy for improving psychomotor development and cognitive function in children under the age of three with iron deficiency anaemia (Cochrane Review)
Quantitative methods in the review of epidemiologic literature
Epidemiol Rev
Antipsychotic-induced weight gain: a comprehensive research systhesis
Am J Psychiatry
Intravenous or enteral loop diuretics for preterm infants with (or developing) chronic lung disease (Cochrane Review)
A method of estimation of missing values in multivariate data suitable for use with an electronic computer
J R Stat Soc
Statistical methods for analysis of effects of treatment in overviews of randomized trials
Nedocromil sodium for preventing exercise-induced bronchoconstriction (Cochrane Review)
The application of statistical methods to industrial standardization and quality control
Massage for promoting growth and development of preterm and/or low birth weight infants (Cochrane Review)
Penicillamine for treating rheumatoid arthritis (Cochrane Review)
Detecting skewness from summary information
Br Med J
Methods for handling missing data in research synthesis
Missing data
Meta-analyses of therapies for postmenopausal osteoporosis. I. Systematic reviews of randomized trials in osteoporosis: Introduction and methodology
Endocr Rev
Cited by (222)
Caffeine consumed prior to cardiac stress testing may affect diagnostic accuracy of nuclear medicine myocardial imaging of myocardial ischemia: A systematic review and meta-analysis
2024, Journal of Medical Imaging and Radiation SciencesResponses of C:N:P stoichiometric correlations among plants, soils and microorganisms to warming: A meta-analysis
2024, Science of the Total EnvironmentResponses of non-structural carbohydrates and biomass in plant to heavy metal treatment
2024, Science of the Total EnvironmentThe role of arbuscular mycorrhizal fungi in the alleviation of cadmium stress in cereals: A multilevel meta-analysis
2023, Science of the Total EnvironmentAnthropogenic soil management performs an important role in increasing soil organic carbon content in northeastern China: A meta-analysis
2023, Agriculture, Ecosystems and Environment