Doing effect size calculations for meta-analysis is a good way to lose your faith in humanity—or at least your faith in researchers’ abilities to do anything like sensible statistical inference. Try it, and you’re surely encounter head-scratchingly weird ways that authors have reported even simple analyses, like basic group comparisons. When you encounter this sort of thing, you have two paths: you can despair, curse, and/or throw things, or you can view the studies as curious little puzzles—brain-teasers, if you will—to keep you awake and prevent you from losing track of those notes you took during your stats courses, back when. Here’s one of those curious little puzzles, which I recently encountered in helping a colleague with a meta-analysis project.
A researcher conducts a randomized experiment, assigning participants to each of \(G\) groups. Each participant is assessed on a variable \(Y\) at pre-test and at post-test (we can assume there’s no attrition). In their study write-up, the researcher reports sample sizes for each group, means and standard deviations for each group at pre-test and at post-test, and adjusted means at post-test, where the adjustment is done using a basic analysis of covariance, controlling for pre-test scores only. The data layout looks like this:
| Group | \(N\) | Pre-test \(M\) | Pre-test \(SD\) | Post-test \(M\) | Post-test \(SD\) | Adjusted post-test \(M\) |
|---|---|---|---|---|---|---|
| Group A | \(n_A\) | \(\bar{x}_{A}\) | \(s_{A0}\) | \(\bar{y}_{A}\) | \(s_{A1}\) | \(\tilde{y}_A\) |
| Group B | \(n_B\) | \(\bar{x}_{B}\) | \(s_{B0}\) | \(\bar{y}_{B}\) | \(s_{B1}\) | \(\tilde{y}_B\) |
| \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) |
Note that the write-up does not provide an estimate of the correlation between the pre-test and the post-test, nor does it report a standard deviation or standard error for the mean change-score between pre-test and post-test within each group. All we have are the summary statistics, plus the adjusted post-test scores. We can assume that the adjustment was done according to the basic ANCOVA model, assuming a common slope across groups as well as homoskedasticity and so on. The model is then \[ y_{ig} = \alpha_g + \beta x_{ig} + e_{ig}, \] for \(i = 1,...,n_g\) and \(g = 1,...,G\), where \(e_{ig}\) is an independent error term that is assumed to have constant variance across groups.
For realz?
Here’s an example with real data, drawn from Table 2 of Murawski (2006):
| Group | \(N\) | Pre-test \(M\) | Pre-test \(SD\) | Post-test \(M\) | Post-test \(SD\) | Adjusted post-test \(M\) |
|---|---|---|---|---|---|---|
| Group A | 25 | 37.48 | 4.64 | 37.96 | 4.35 | 37.84 |
| Group B | 26 | 36.85 | 5.18 | 36.46 | 3.86 | 36.66 |
| Group C | 16 | 37.88 | 3.88 | 37.38 | 4.76 | 36.98 |
That study reported this information for each of several outcomes, with separate analyses for each of two sub-groups (LD and NLD). The text also reports that they used a two-level hierarchical linear model for the ANCOVA adjustment. For simplicity, let’s just ignore the hierarchical linear model aspect and assume that it’s a straight, one-level ANCOVA.
The puzzler
Calculate an estimate of the standardized mean difference between group \(B\) and group \(A\), along with the sampling variance of the SMD estimate, that adjusts for pre-test differences between groups. Candidates for numerator of the SMD include the adjusted mean difference, \(\tilde{y}_B - \tilde{y}_A\) or the difference-in-differences, \(\left(\bar{y}_B - \bar{x}_B\right) - \left(\bar{y}_A - \bar{x}_A\right)\). In either case, the tricky bit is finding the sampling variance of this quantity, which involves the pre-post correlation. For the denominator of the SMD, you use the post-test SD, either pooled across just groups \(A\) and \(B\) or pooled across all \(G\) groups, assuming a common population variance.
Have an idea for how to solve this? Post it in the comments or email it to me. Need the solution because you have a study like this in your meta-analysis? Contact me and I’ll share it with you directly. I’m being coy because I’m teaching meta-analysis next semester, and I feel like this would make a good extra credit problem…