学校层次比较中断时间序列设计的内部有效性——来自四个新的研究内比较的证据

IF 1.7 4区教育学 Q2 EDUCATION & EDUCATIONAL RESEARCH

Journal of Research on Educational Effectiveness Pub Date : 2022-04-15 DOI:10.1080/19345747.2022.2051652

Sam Sims, Jake Anders, Laura Zieger

{"title":"学校层次比较中断时间序列设计的内部有效性——来自四个新的研究内比较的证据","authors":"Sam Sims, Jake Anders, Laura Zieger","doi":"10.1080/19345747.2022.2051652","DOIUrl":null,"url":null,"abstract":"Abstract Comparative interrupted time series (CITS) designs evaluate impact by modeling the relative deviation from trends among a treatment and comparison group after an intervention. The broad applicability of the design means it is widely used in education research. Like all non-experimental evaluation methods however, the internal validity of a given CITS evaluation depends on assumptions that cannot be directly verified. We provide an empirical test of the internal validity of CITS by conducting four within-study comparisons of school-level interventions previously evaluated using randomized controlled trials. Our estimate of bias across these four studies is 0.03 school-level (or 0.01 pupil-level) standard deviations. The results suggest well-conducted CITS evaluations of similar school-level education interventions are likely to display limited bias.","PeriodicalId":47260,"journal":{"name":"Journal of Research on Educational Effectiveness","volume":"15 1","pages":"876 - 897"},"PeriodicalIF":1.7000,"publicationDate":"2022-04-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"6","resultStr":"{\"title\":\"The Internal Validity of the School-Level Comparative Interrupted Time Series Design: Evidence From Four New Within-Study Comparisons\",\"authors\":\"Sam Sims, Jake Anders, Laura Zieger\",\"doi\":\"10.1080/19345747.2022.2051652\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Abstract Comparative interrupted time series (CITS) designs evaluate impact by modeling the relative deviation from trends among a treatment and comparison group after an intervention. The broad applicability of the design means it is widely used in education research. Like all non-experimental evaluation methods however, the internal validity of a given CITS evaluation depends on assumptions that cannot be directly verified. We provide an empirical test of the internal validity of CITS by conducting four within-study comparisons of school-level interventions previously evaluated using randomized controlled trials. Our estimate of bias across these four studies is 0.03 school-level (or 0.01 pupil-level) standard deviations. The results suggest well-conducted CITS evaluations of similar school-level education interventions are likely to display limited bias.\",\"PeriodicalId\":47260,\"journal\":{\"name\":\"Journal of Research on Educational Effectiveness\",\"volume\":\"15 1\",\"pages\":\"876 - 897\"},\"PeriodicalIF\":1.7000,\"publicationDate\":\"2022-04-15\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"6\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of Research on Educational Effectiveness\",\"FirstCategoryId\":\"95\",\"ListUrlMain\":\"https://doi.org/10.1080/19345747.2022.2051652\",\"RegionNum\":4,\"RegionCategory\":\"教育学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"EDUCATION & EDUCATIONAL RESEARCH\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Research on Educational Effectiveness","FirstCategoryId":"95","ListUrlMain":"https://doi.org/10.1080/19345747.2022.2051652","RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 6

摘要

摘要比较中断时间序列（CITS）设计通过建模干预后治疗组和对照组与趋势的相对偏差来评估影响。该设计的广泛适用性意味着它在教育研究中得到了广泛应用。然而，与所有非实验评估方法一样，给定CITS评估的内部有效性取决于无法直接验证的假设。我们通过对先前使用随机对照试验评估的学校层面干预措施进行四次研究内比较，对CITS的内部有效性进行了实证检验。我们对这四项研究的偏差估计为0.03学校水平（或0.01学生水平）的标准差。研究结果表明，CITS对类似学校层面的教育干预措施进行的良好评估可能显示出有限的偏见。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

The Internal Validity of the School-Level Comparative Interrupted Time Series Design: Evidence From Four New Within-Study Comparisons

Abstract Comparative interrupted time series (CITS) designs evaluate impact by modeling the relative deviation from trends among a treatment and comparison group after an intervention. The broad applicability of the design means it is widely used in education research. Like all non-experimental evaluation methods however, the internal validity of a given CITS evaluation depends on assumptions that cannot be directly verified. We provide an empirical test of the internal validity of CITS by conducting four within-study comparisons of school-level interventions previously evaluated using randomized controlled trials. Our estimate of bias across these four studies is 0.03 school-level (or 0.01 pupil-level) standard deviations. The results suggest well-conducted CITS evaluations of similar school-level education interventions are likely to display limited bias.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

Journal of Research on Educational Effectiveness EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

4.00

自引率

11.10%

发文量

期刊介绍： As the flagship publication for the Society for Research on Educational Effectiveness, the Journal of Research on Educational Effectiveness (JREE) publishes original articles from the multidisciplinary community of researchers who are committed to applying principles of scientific inquiry to the study of educational problems. Articles published in JREE should advance our knowledge of factors important for educational success and/or improve our ability to conduct further disciplined studies of pressing educational problems. JREE welcomes manuscripts that fit into one of the following categories: (1) intervention, evaluation, and policy studies; (2) theory, contexts, and mechanisms; and (3) methodological studies. The first category includes studies that focus on process and implementation and seek to demonstrate causal claims in educational research. The second category includes meta-analyses and syntheses, descriptive studies that illuminate educational conditions and contexts, and studies that rigorously investigate education processes and mechanism. The third category includes studies that advance our understanding of theoretical and technical features of measurement and research design and describe advances in data analysis and data modeling. To establish a stronger connection between scientific evidence and educational practice, studies submitted to JREE should focus on pressing problems found in classrooms and schools. Studies that help advance our understanding and demonstrate effectiveness related to challenges in reading, mathematics education, and science education are especially welcome as are studies related to cognitive functions, social processes, organizational factors, and cultural features that mediate and/or moderate critical educational outcomes. On occasion, invited responses to JREE articles and rejoinders to those responses will be included in an issue.