Discussion
Missing variance measures are a prevalent problem in research synthesis.
Yet, few ecological meta-analyses have adapted imputation algorithms to
handle missing values (Fig. 1). Our study demonstrates how the omission
of incompletely reported studies (complete-case analysis), generally
increases the confidence intervals and how it results in deviating
(potentially even biased) grand mean estimates if SDs/SSs are not
missing completely at random. The R-code used to simulate and compare
the effects of different meta-analysis datasets structures, patterns of
missingness and options to handle missing data is freely available at
github.com/StephanKambach/SimulateMissingDataInMeta-Analyses. Although
our number of ten replicates is at the lower end of the desired
replications in simulation studies, it was enough to show the general
effects of treating missing SDs and SSs and meta-analysis data sets.
In accordance with previous publications, we found that unweighted
analyses yielded grand mean estimates that were unbiased with regard to
fully informed weighted analyses as long as effect sizes and their
corresponding variance estimates were normally and independently
distributed. The same holds for sample-size-approximated effect sizes
variances. In case of a potential relationship between effect sizes and
effect size precision (maybe due to different study designs) we advise
to apply imputation methods to fill missing SDs and/or SSs.
If SDs and/or SSs are both MCAR and unrelated to effect sizes, the
imputation of up to 90% of missing data yielded grand means similar to
those obtained from fully informed weighted meta-analyses. Below a
threshold of ca. 50-60% of missing SDs and/or SSs, imputation methods
performed equally or outperformed complete-case, unweighted and
sample-size weighted analyses. Yet, our results also demonstrated, that
different imputation methods can accommodate different dataset
structures regarding missingness and correlation patterns. Mean, median
and random sample imputations are easy to implement but biased in case
of a relationship between effect sizes and effect size precision.
Methods applying predictive mean matching tend to suit such
relationships but tend to yield a larger confidence intervals of the
grand mean. Thus, for any meta-analysis, the method used to deal with
missing SDs and/or SSs should be chosen under the following
considerations: