Journal of Research on Educational Effectiveness最新文献_第9页

Propensity Score Methods and Difference-in-Differences with an Exogenous Time-Varying Confounder: Evaluation of Methods 外生时变混杂因素的倾向评分方法和差异中的差异:方法的评价

IF 1.8 4区教育学

Journal of Research on Educational Effectiveness Pub Date : 2022-11-01 DOI: 10.1080/19345747.2022.2128485

Peter Boedeker

{"title":"Propensity Score Methods and Difference-in-Differences with an Exogenous Time-Varying Confounder: Evaluation of Methods","authors":"Peter Boedeker","doi":"10.1080/19345747.2022.2128485","DOIUrl":"https://doi.org/10.1080/19345747.2022.2128485","url":null,"abstract":"Abstract Quasi-experimental designs (QEDs) are used to estimate a treatment effect without randomization. Confounders have a causal relationship with the outcome and probability of treatment adoption and if unaccounted for can bias treatment effect estimates. A variable considered a confounder prior to treatment can change after treatment has occurred (i.e., a time-varying confounder) not as a result of treatment (what we call an exogenous time-varying confounder). If the post-treatment value causally affects the outcome to change and this post-treatment value of the exogenous time-varying confounder is unaccounted for, then the treatment effect may be biased. We review the Rubin Causal Model and QED assumptions and the effect an exogenous time-varying confounder has on the ability of QEDs to produce an appropriate counterfactual. We conduct a simulation study evaluating propensity score and difference-in-differences based methods for estimating a treatment effect with an exogenous time-varying confounder. Propensity score weighted two-way fixed effects, inverse probability weighted, or doubly robust difference-in-differences methods, each with propensity scores estimated using post-implementation values of the exogenous time-varying confounder, proved least biased when the exogenous time-varying confounder changed differentially for members of the treatment and control groups.","PeriodicalId":47260,"journal":{"name":"Journal of Research on Educational Effectiveness","volume":null,"pages":null},"PeriodicalIF":1.8,"publicationDate":"2022-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46664319","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

A Cautionary Tale of Tutoring Hard-to-Reach Students in Kenya 在肯尼亚辅导难以接触的学生的警示故事

IF 1.8 4区教育学

Journal of Research on Educational Effectiveness Pub Date : 2022-11-01 DOI: 10.1080/19345747.2022.2131661

Beth E. Schueler, Daniel Rodriguez-Segura

{"title":"A Cautionary Tale of Tutoring Hard-to-Reach Students in Kenya","authors":"Beth E. Schueler, Daniel Rodriguez-Segura","doi":"10.1080/19345747.2022.2131661","DOIUrl":"https://doi.org/10.1080/19345747.2022.2131661","url":null,"abstract":"Abstract Covid-19 school closures generated interest in tutoring to make up for lost learning time. Tutoring is backed by rigorous research, but it is unclear whether it can be delivered effectively remotely. We study the effect of teacher-student phone calls in Kenya when schools were closed. Schools (n = 105) were randomly assigned for 3rd, 5th and 6th graders (n = 8,319) to receive one of two versions of a 7-week weekly math intervention—5-minute accountability checks or 15-min mini-tutoring sessions—or to the control group. Although calls increased perceptions that teachers cared, accountability checks had no effect on math performance four months later and tutoring decreased achievement among students who returned to their schools after reopening. This was, in part, because the relatively low-achieving students most likely to benefit from calls were least likely to return and take assessments. Tutoring substituted away from more productive uses of time, at least among returners. Neither intervention affected enrollment. Tutoring remains a valuable tool but to avoid unintended consequences, careful attention should be paid to aligning interventions with best practices and targeting to those who benefit most.","PeriodicalId":47260,"journal":{"name":"Journal of Research on Educational Effectiveness","volume":null,"pages":null},"PeriodicalIF":1.8,"publicationDate":"2022-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48319071","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 11

Effects of Cross-Age Peer Mentoring Program within a Randomized Controlled Trial 一项随机对照试验中跨年龄同伴辅导计划的效果

IF 1.8 4区教育学

Journal of Research on Educational Effectiveness Pub Date : 2022-11-01 DOI: 10.1080/19345747.2022.2130119

E. Jenner, Katherine Lass, Sarah L. Walsh, H. Demby, Rebekah Leger, Gretchen Falk

{"title":"Effects of Cross-Age Peer Mentoring Program within a Randomized Controlled Trial","authors":"E. Jenner, Katherine Lass, Sarah L. Walsh, H. Demby, Rebekah Leger, Gretchen Falk","doi":"10.1080/19345747.2022.2130119","DOIUrl":"https://doi.org/10.1080/19345747.2022.2130119","url":null,"abstract":"Abstract This paper summarizes results from an impact study that employed a randomized controlled trial to estimate the efficacy of a cross-age peer mentor program designed to prevent school dropout during the transition from middle to high school. We present findings from the intent-to-treat (ITT) analyses, which included 1,351 ninth-grade students, alongside those of two different methods that estimate the complier average causal effect (CACE) of participating in the program. Although the confirmatory study, which investigated impact on attendance and credit accrual in ninth grade, was null, ITT analyses on exploratory outcomes indicate modest, yet potentially meaningful program impact on ninth-grade outcomes of discipline, school attachment, and expectations of degree attainment across varying dosage levels. CACE estimates also suggest that a threshold level of program participation broadens the program’s impact on additional exploratory academic achievement and social and emotional learning outcomes. Given the adverse effects of the transition to high school, this promising evidence indicates that the cross-age peer mentoring intervention could be an effective strategy for high schools to implement that leverages existing staff and students.","PeriodicalId":47260,"journal":{"name":"Journal of Research on Educational Effectiveness","volume":null,"pages":null},"PeriodicalIF":1.8,"publicationDate":"2022-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45733877","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Evaluation of a State-Wide Mathematics Support Program for at-Risk Students in Grade 1 and 2 in Germany 德国一年级和二年级高危学生全国数学支持计划的评价

IF 1.8 4区教育学

Journal of Research on Educational Effectiveness Pub Date : 2022-10-02 DOI: 10.1080/19345747.2022.2051651

Ann-Katrin van den Ham, Aiso Heinze

{"title":"Evaluation of a State-Wide Mathematics Support Program for at-Risk Students in Grade 1 and 2 in Germany","authors":"Ann-Katrin van den Ham, Aiso Heinze","doi":"10.1080/19345747.2022.2051651","DOIUrl":"https://doi.org/10.1080/19345747.2022.2051651","url":null,"abstract":"Abstract Supporting students with difficulties in learning mathematics is a challenge for teachers and educational administrators. Formative assessment is considered to play a successful role in supporting at-risk students as well as students without difficulties in mathematics. There is a need for intervention programs, including formative assessment techniques, that (a) are easy to implement in the regular classroom without requiring radical changes in teachers’ individual teaching style, and (b) are effective in supporting at-risk students at the earliest stage possible in their school careers. This article analyzes an effectiveness trial of a formative assessment program developed to meet these goals and conducted in the first two years of elementary school. The examination of the longitudinal dataset from Grades 1–3 (N = 2,330) revealed an effect after the implementation, which was maintained at nearly the same effect size one year after completion of the program. The findings imply that formative assessment can foster the arithmetic achievement of students at risk as well as that of the entire class without changing the curriculum or teachers’ individual teaching style.","PeriodicalId":47260,"journal":{"name":"Journal of Research on Educational Effectiveness","volume":null,"pages":null},"PeriodicalIF":1.8,"publicationDate":"2022-10-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42378193","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Covariate Balance for Observational Effectiveness Studies: A Comparison of Matching and Weighting 观察有效性研究的协变量平衡:匹配和加权的比较

IF 1.8 4区教育学

Journal of Research on Educational Effectiveness Pub Date : 2022-09-07 DOI: 10.1080/19345747.2022.2110545

Joseph M. Kush, Elise T. Pas, R. Musci, Catherine P. Bradshaw

{"title":"Covariate Balance for Observational Effectiveness Studies: A Comparison of Matching and Weighting","authors":"Joseph M. Kush, Elise T. Pas, R. Musci, Catherine P. Bradshaw","doi":"10.1080/19345747.2022.2110545","DOIUrl":"https://doi.org/10.1080/19345747.2022.2110545","url":null,"abstract":"Abstract Propensity score matching and weighting methods are often used in observational effectiveness studies to reduce imbalance between treated and untreated groups on a set of potential confounders. However, much of the prior methodological literature on matching and weighting has yet to examine performance for scenarios with a majority of treated units, as is often encountered with programs and interventions that have been widely disseminated or “scaled-up.” Using a series of Monte Carlo simulations, we compare the performance of k:1 matching with replacement and weighting methods with respect to covariate balance, bias, and mean squared error. Results indicate that the accuracy of all methods declined as treatment prevalence increased. While weighting produced the largest reduction in covariate imbalance, 1:1 matching with replacement provided the most unbiased treatment effect estimates. An applied example using empirical school-level data is provided to further illustrate the application and interpretation of these methods to a real-world scale-up effort. We conclude by considering the implications of propensity score methods for observational effectiveness studies with a particular focus on educational research.","PeriodicalId":47260,"journal":{"name":"Journal of Research on Educational Effectiveness","volume":null,"pages":null},"PeriodicalIF":1.8,"publicationDate":"2022-09-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43304081","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Using Robust Standard Errors for the Analysis of Binary Outcomes with a Small Number of Clusters 基于鲁棒标准误差的小簇二值结果分析

IF 1.8 4区教育学

Journal of Research on Educational Effectiveness Pub Date : 2022-08-08 DOI: 10.1080/19345747.2022.2100301

Francis L. Huang, Bixiu Zhang, Xintong Li

引用次数: 2

Making Sense of Effect Sizes: Systematic Differences in Intervention Effect Sizes by Outcome Measure Type 理解效果大小：按结果测量类型划分的干预效果大小的系统差异

IF 1.8 4区教育学

Journal of Research on Educational Effectiveness Pub Date : 2022-08-01 DOI: 10.1080/19345747.2022.2071364

Betsy Wolf, Erica Harbatkin

{"title":"Making Sense of Effect Sizes: Systematic Differences in Intervention Effect Sizes by Outcome Measure Type","authors":"Betsy Wolf, Erica Harbatkin","doi":"10.1080/19345747.2022.2071364","DOIUrl":"https://doi.org/10.1080/19345747.2022.2071364","url":null,"abstract":"Abstract One challenge in understanding “what works” in education is that effect sizes may not be comparable across studies, raising questions for practitioners and policymakers using research to select interventions. One factor that consistently relates to the magnitude of effect sizes is the type of outcome measure. This article uses study data from the What Works Clearinghouse to determine average effect sizes by outcome measure type. Outcome measures were categorized by whether the group who developed the measure potentially had a stake in the intervention (non-independent) or not (independent). Using meta-analysis and controlling for study quality and intervention characteristics, we find larger average effect sizes for non-independent measures than for independent measures. Results suggest that larger effect sizes for non-independent measures are not due to differences in implementation fidelity, study quality, or intervention or sample characteristics. Instead, non-independent and independent measures appear to represent partially but minimally overlapping latent constructs. Findings call into question whether policymakers and practitioners should make decisions based on non-independent measures when they are ultimately responsible for improving outcomes on independent measures.","PeriodicalId":47260,"journal":{"name":"Journal of Research on Educational Effectiveness","volume":null,"pages":null},"PeriodicalIF":1.8,"publicationDate":"2022-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49353661","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 9

Raising Teacher Retention in Online Courses through Personalized Support. Evidence from a Cross-National Randomized Controlled Trial 通过个性化支持提高在线课程中的教师保留率。来自一项跨国家随机对照试验的证据

IF 1.8 4区教育学

Journal of Research on Educational Effectiveness Pub Date : 2022-07-26 DOI: 10.1080/19345747.2022.2100850

D. Azzolini, Sonia Marzadro, E. Rettore, Katja Engelhardt, Benjamin Hertz, Patricia Wastiau

{"title":"Raising Teacher Retention in Online Courses through Personalized Support. Evidence from a Cross-National Randomized Controlled Trial","authors":"D. Azzolini, Sonia Marzadro, E. Rettore, Katja Engelhardt, Benjamin Hertz, Patricia Wastiau","doi":"10.1080/19345747.2022.2100850","DOIUrl":"https://doi.org/10.1080/19345747.2022.2100850","url":null,"abstract":"Abstract Online courses have become an increasingly attractive format for delivering teacher training. However, the low retention rates are a critical and still unsolved issue. This paper presents the results of a randomized controlled trial aimed at testing the impact of a personalized support model on teachers’ retention in online training courses. The support consisted of a package of nine messages triggered by teachers’ characteristics and their specific (in)actions on the course platform. The study involved 3,777 lower-secondary education professional and student teachers from nine European Union Member States and Turkey, who were invited to participate in four new online courses in school year 2018/2019. The experimental estimates show that the offered support increased course completion by 10 percentage points among professional teachers in EU Member States, while it had no effects among student teachers nor in Turkey. Implications for online teacher training providers—such as the importance of reaching out to participants with poor online training experience and who do not start the courses in time—are discussed.","PeriodicalId":47260,"journal":{"name":"Journal of Research on Educational Effectiveness","volume":null,"pages":null},"PeriodicalIF":1.8,"publicationDate":"2022-07-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45445821","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 3

Examining the Impact of a First Grade Whole Number Intervention by Group Size 按小组规模检验一年级整数干预的影响

IF 1.8 4区教育学

Journal of Research on Educational Effectiveness Pub Date : 2022-07-22 DOI: 10.1080/19345747.2022.2093299

Ben Clarke, Christian T. Doabler, Marah Sutherland, Derek B. Kosty, Jessica E. Turtura, K. Smolkowski

{"title":"Examining the Impact of a First Grade Whole Number Intervention by Group Size","authors":"Ben Clarke, Christian T. Doabler, Marah Sutherland, Derek B. Kosty, Jessica E. Turtura, K. Smolkowski","doi":"10.1080/19345747.2022.2093299","DOIUrl":"https://doi.org/10.1080/19345747.2022.2093299","url":null,"abstract":"Abstract This study utilized a partially nested randomized control design to investigate the impact of Fusion, a first-grade math intervention. Blocking on classrooms, students were randomly assigned to one of the three conditions: a Fusion two-student group, a Fusion five student group, or a no-treatment control group. Two primary research questions were examined: What was the overall impact of the Fusion intervention as compared to a business-as-usual comparison condition? and Was there a differential impact on student outcomes between the 2:1 Fusion and the 5:1 Fusion conditions? Analyses found a positive effects on four outcome measures favoring Fusion groups over control with two of the differences statistically significant. Results between Fusion groups found positive effects favoring the Fusion 2:1 group compared to the Fusion 5:1 group on all four outcome measures with two of the differences statistically significant. On a second-grade follow-up measure, no difference was found between Fusion groups and control, but a statistically significant difference was found between Fusion groups favoring the 2:1 Fusion group. Future research directions and implications for practice are discussed.","PeriodicalId":47260,"journal":{"name":"Journal of Research on Educational Effectiveness","volume":null,"pages":null},"PeriodicalIF":1.8,"publicationDate":"2022-07-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42531601","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Quantifying “Promising Trials Bias” in Randomized Controlled Trials in Education 在教育随机对照试验中量化“有希望的试验偏差”

IF 1.8 4区教育学

Journal of Research on Educational Effectiveness Pub Date : 2022-07-11 DOI: 10.1080/19345747.2022.2090470

Sam Sims, Jake Anders, Matthew Inglis, Hugues Lortie-Forgues

引用次数: 0