New working paper: Procedural sensitivities of SCD effect sizes

James E. Pustejovsky

I’ve just posted a new version of my working paper, Procedural sensitivities of effect sizes for single-case designs with behavioral outcome measures. The abstract is below. This version is a major update of an earlier paper that focused only on the non-overlap measures. The new version also includes analysis of two other effect sizes (the within-case standardized mean difference and the log response ratio) as well as additional results and more succinct summaries of the main findings.

The paper itself is available on the Open Science Framework (here), as are the supplementary materials and Source code. I also created interaction versions of the graphics from the main paper and the supplementary materials, which can be viewed in this shiny app. I would welcome any comments, questions, or feedback that readers may have.

Abstract

A wide variety of effect size indices have been proposed for quantifying the magnitude of treatment effects in single-case designs. Commonly used measures include parametric indices such as the standardized mean difference, as well as non-overlap measures, such as the percentage of non-overlapping data, improvement rate difference, and non-overlap of all pairs. Currently, little is known about the properties of these indices when applied to behavioral data collected by systematic direct observation, even though systematic direct observation is the most common approach to outcome measurement in single-case research. This study uses computer simulation to investigate the properties of several single-case effect size measures when applied to systematic direct observation data. Results indicate that the magnitude of the non-overlap measures and of the standardized mean difference can be strongly influenced by procedural details of the study’s design, which is a significant limitation to using these indices as effect sizes for meta-analysis of single-case designs. A less widely used parametric index, the log-response ratio, has the advantage of being insensitive to sample size and observation session length, although its magnitude is influenced by the use of partial interval recording.

--- title: 'New working paper: Procedural sensitivities of SCD effect sizes' date: '2016-10-17' categories: - effect size - single-case design - response ratio - non-overlap measures - simulation code-tools: true --- I've just posted a new version of my working paper, _Procedural sensitivities of effect sizes for single-case designs with behavioral outcome measures_. The abstract is below. This version is a major update of an [earlier paper](/files/Pustejovsky-2015-Nov-Non-overlap-measures.pdf) that focused only on the non-overlap measures. The new version also includes analysis of two other effect sizes (the within-case standardized mean difference and the log response ratio) as well as additional results and more succinct summaries of the main findings. The paper itself is available on the Open Science Framework ([here](https://osf.io/pxn24/)), as are the [supplementary materials](https://osf.io/hkzsm/) and [Source code](https://osf.io/j4gvt/). I also created interaction versions of the graphics from the main paper and the supplementary materials, which can be viewed in [this shiny app](https://jepusto.shinyapps.io/SCD-effect-size-sensitivities/). I would welcome any comments, questions, or feedback that readers may have. ### Abstract > A wide variety of effect size indices have been proposed for quantifying the magnitude of treatment effects in single-case designs. Commonly used measures include parametric indices such as the standardized mean difference, as well as non-overlap measures, such as the percentage of non-overlapping data, improvement rate difference, and non-overlap of all pairs. Currently, little is known about the properties of these indices when applied to behavioral data collected by systematic direct observation, even though systematic direct observation is the most common approach to outcome measurement in single-case research. This study uses computer simulation to investigate the properties of several single-case effect size measures when applied to systematic direct observation data. Results indicate that the magnitude of the non-overlap measures and of the standardized mean difference can be strongly influenced by procedural details of the study's design, which is a significant limitation to using these indices as effect sizes for meta-analysis of single-case designs. A less widely used parametric index, the log-response ratio, has the advantage of being insensitive to sample size and observation session length, although its magnitude is influenced by the use of partial interval recording.