The Relationship between Intercoder Reliability of Data Extraction and Effect Measure Calculation in Single-Case Meta-Analysis

Abstract

AbstractMeta-analyzing data from single-case experimental designs (SCEDs) usually requires data extraction, a process by which numerical values are obtained from linear graphs in primary studies, prior to calculating and aggregating single-case effect measures. Existing research suggests data extraction yields reliable and valid data; however, we have an incomplete understanding of the downstream effects of relying on data extracted by two or more people. This study was undertaken to enhance that understanding in the context of SCEDs published in school psychology journals. Data for 91 unique outcomes across 67 cases in 20 SCEDs were extracted by two data extractors. Four different single-case effect measures were calculated using data extracted by each data extractor and then compared to determine the similarity of the effect measures. Overall, intercoder reliability metrics suggested a high degree of agreement, and there were minimal differences in single-case effect measures calculated from data extracted by different researchers. Intercoder reliability metrics and differences in single-case effect measures were generally negatively related, though the strength varied depending on the single-case effect measure. Hence, it is unlikely that the small differences in effect measure estimates due to the slight unreliability of the data extraction process would have a considerable impact on the interpretation of single-case effect measures.Impact StatementTwo people extracted highly similar numerical data from the same linear graphs using plot digitizing software. Differences in calculations across data extracted by two people were trivial. Results suggest researchers can likely have confidence in the calculation of effect measures aggregated in meta-analyses of single-case experimental designs, provided they achieve comparable levels of agreement amongst data extractors.Keywords: single subject designsmeta-analysisresearch methodsASSOCIATE EDITOR: Jorge E. Gonzalez DISCLOSUREThe authors have no conflicts of interest to report.Open ScholarshipThis article has earned the Center for Open Science badges for Open Data and Open Materials through Open Practices Disclosure. The data and materials are openly accessible at https://osf.io/249w7/ and https://osf.io/249w7/. To obtain the author's disclosure form, please contact the Editor.Additional informationFundingThis study was supported by the Faculty Research and Creative Endeavors committee at Central Michigan University.Notes on contributorsDaniel D. DrevonDaniel D. Drevon, PhD, is an Associate Professor and Program Director with the School Psychology Program at Central Michigan University. He is interested in academic and behavioral interventions, single-case experimental design, and research synthesis/meta-analysis.Allison M. PeartAllison M. Peart, MA, is a doctoral candidate in the School Psychology Program at Central Michigan University and predoctoral intern at the University of Nebraska Medical Center’s Munroe-Meyer Institute. She is interested in behavioral interventions, single-case experimental design, and research synthesis/meta-analysis.Elizabeth T. KovalElizabeth T. Koval, MA, is a doctoral candidate in the School Psychology Program at Central Michigan University. She is interested in behavioral interventions and teacher consultation.