The Design of Making Pre-K count and High Fives: Two-Stage, Multiyear Random Assignment at Different Levels.

IF 3 4区 社会学 Q1 SOCIAL SCIENCES, INTERDISCIPLINARY
Evaluation Review Pub Date : 2023-08-01 Epub Date: 2023-03-04 DOI:10.1177/0193841X231159472
Shira Mattera, Marie-Andree Somers, Robin Jacob, Pamela Morris-Perez
{"title":"The Design of Making Pre-K count and High Fives: Two-Stage, Multiyear Random Assignment at Different Levels.","authors":"Shira Mattera, Marie-Andree Somers, Robin Jacob, Pamela Morris-Perez","doi":"10.1177/0193841X231159472","DOIUrl":null,"url":null,"abstract":"<p><p>The Making Pre-K Count and High 5s studies represent a recent application of a phased two-stage, multi-level design, which was used to examine the effects of two aligned math programs implemented in early childhood settings. The purpose of this paper is to describe the challenges encountered in implementing this two-stage design and to describe approaches to resolving them. We then present a set of sensitivity analyses the study team used to examine the robustness of the findings. During the pre-K year, pre-K <i>centers</i> were randomly assigned either to receive an evidence-based early math curriculum and associated professional development (Making Pre-K Count) or to a pre-K-as-usual control condition. In the kindergarten year, <i>students</i> who had been in Making Pre-K Count program classrooms in pre-K were then individually randomly assigned within schools to small-group supplemental math clubs that were designed to sustain the gains from the pre-K program, or to a business-as-usual kindergarten experience. Making Pre-K Count took place in 69 pre-K sites, comprising 173 classrooms across New York City. High 5s took place in the 24 sites that were part of the public school treatment arm of the Making Pre-K Count study and included 613 students. The study focuses on the effect of the Making Pre-K Count and High 5s programs on children's math skills at the end of kindergarten as measured by two instruments, the Research-Based Early Math Assessment-Kindergarten (REMA-K) and the Woodcock-Johnson Applied Problems test. The multi-armed design, while logistically and analytically challenging, balanced multiple considerations of power, the number of research questions that could be answered, and efficiency of resources. Robustness checks suggest that the design created groups that were both meaningfully and statistically equivalent. Decisions to use a phased multi-armed design should consider both its strengths and weaknesses. While the design allows for a more flexible, expansive research study, it also introduces complexities that need to be addressed both logistically and analytically.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"47 4","pages":"701-726"},"PeriodicalIF":3.0000,"publicationDate":"2023-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Evaluation Review","FirstCategoryId":"90","ListUrlMain":"https://doi.org/10.1177/0193841X231159472","RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2023/3/4 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"SOCIAL SCIENCES, INTERDISCIPLINARY","Score":null,"Total":0}
引用次数: 0

Abstract

The Making Pre-K Count and High 5s studies represent a recent application of a phased two-stage, multi-level design, which was used to examine the effects of two aligned math programs implemented in early childhood settings. The purpose of this paper is to describe the challenges encountered in implementing this two-stage design and to describe approaches to resolving them. We then present a set of sensitivity analyses the study team used to examine the robustness of the findings. During the pre-K year, pre-K centers were randomly assigned either to receive an evidence-based early math curriculum and associated professional development (Making Pre-K Count) or to a pre-K-as-usual control condition. In the kindergarten year, students who had been in Making Pre-K Count program classrooms in pre-K were then individually randomly assigned within schools to small-group supplemental math clubs that were designed to sustain the gains from the pre-K program, or to a business-as-usual kindergarten experience. Making Pre-K Count took place in 69 pre-K sites, comprising 173 classrooms across New York City. High 5s took place in the 24 sites that were part of the public school treatment arm of the Making Pre-K Count study and included 613 students. The study focuses on the effect of the Making Pre-K Count and High 5s programs on children's math skills at the end of kindergarten as measured by two instruments, the Research-Based Early Math Assessment-Kindergarten (REMA-K) and the Woodcock-Johnson Applied Problems test. The multi-armed design, while logistically and analytically challenging, balanced multiple considerations of power, the number of research questions that could be answered, and efficiency of resources. Robustness checks suggest that the design created groups that were both meaningfully and statistically equivalent. Decisions to use a phased multi-armed design should consider both its strengths and weaknesses. While the design allows for a more flexible, expansive research study, it also introduces complexities that need to be addressed both logistically and analytically.

让学前班发挥作用和击掌庆祝活动的设计:不同级别的两阶段、多年期随机分配。
让学前班发挥重要作用 "和 "High 5s 研究 "是分阶段、两阶段、多层次设计的最新应用。本文旨在描述在实施这种两阶段设计时遇到的挑战,并介绍解决这些挑战的方法。然后,我们介绍了研究小组用来检验研究结果稳健性的一系列敏感性分析。在学前班学年,学前班中心被随机分配到接受循证早期数学课程和相关专业发展("学前班有价值")或学前班照常对照组。在幼儿园学年,曾经在 "学前班有价值 "项目教室学习过学前班课程的学生被随机分配到学校内的小组补充数学俱乐部,该俱乐部旨在维持学前班课程的成果,或照常上课。让学前班更有价值 "活动在纽约市 69 个学前班学校进行,共有 173 间教室。High 5s 在 "学前班有价值 "研究的公立学校治疗部分的 24 个地点进行,包括 613 名学生。研究的重点是 "学前班有价值 "和 "High 5s 计划 "对儿童在幼儿园结束时数学技能的影响,这两种测试工具分别是 "基于研究的早期数学评估--幼儿园"(REMA-K)和 "伍德科克-约翰逊应用问题测试"(Woodcock-Johnson Applied Problems Test)。虽然多臂设计在后勤和分析方面具有挑战性,但它平衡了多方面的考虑,包括力量、可回答的研究问题数量和资源效率。稳健性检验表明,该设计所创建的小组在意义上和统计上都是等同的。决定使用分阶段多臂设计时,应同时考虑其优点和缺点。虽然这种设计允许进行更灵活、更广泛的研究,但它也带来了在后勤和分析方面需要解决的复杂问题。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Evaluation Review
Evaluation Review SOCIAL SCIENCES, INTERDISCIPLINARY-
CiteScore
2.90
自引率
11.10%
发文量
80
期刊介绍: Evaluation Review is the forum for researchers, planners, and policy makers engaged in the development, implementation, and utilization of studies aimed at the betterment of the human condition. The Editors invite submission of papers reporting the findings of evaluation studies in such fields as child development, health, education, income security, manpower, mental health, criminal justice, and the physical and social environments. In addition, Evaluation Review will contain articles on methodological developments, discussions of the state of the art, and commentaries on issues related to the application of research results. Special features will include periodic review essays, "research briefs", and "craft reports".
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信