Model-Building Considerations in Meta-Analysis of Dependent Effect Sizes

In fields ranging from Education to Economics to Ecology, meta-analysts often encounter complicated data structures, in which some or all primary studies include multiple effect size estimates. These estimates may be correlated because they are based on data from a common sample or a partially overlapping sample, or may be statistically dependent due to use of common study operations. A broad analytic strategy for dealing with such data is to specify a “working model” to roughly characterize the dependence structure, then use robust inference strategies that work well even if the working model is mis-specified relative to the true data-generating process. Although the technical and computational aspects of this strategy are now well developed, questions remain about how to apply it effectively in practice. In this talk, I will examine two practical questions related to how to build models for meta-analyses involving dependent effect sizes. First, I will illustrate some connections between working models and simpler, ad hoc techniques for dealing with effect size multiplicity, arguing that these connections provide useful heuristics to guide specification of random effects structures in multi-level and multi-variate meta-analysis. Second, I will describe some analytic strategies for conducting equity-related moderator analyses, where predictors involve personal characteristics of the primary study participants that can vary both within and between studies. I distinguish between direct evidence and contextual evidence about equity of impacts and show that the choice of working model can be consequential for analyses involving direct evidence. Throughout, I will highlight some open issues and practical challenges involved in modeling dependent effect sizes.