Joshua B. Gilbert, Zachary Himmelsbach, James Soland, Mridul Joshi, Benjamin W. Domingue
{"title":"Estimating heterogeneous treatment effects with item-level outcome data: Insights from Item Response Theory","authors":"Joshua B. Gilbert, Zachary Himmelsbach, James Soland, Mridul Joshi, Benjamin W. Domingue","doi":"10.1002/pam.70025","DOIUrl":"https://doi.org/10.1002/pam.70025","url":null,"abstract":"Analyses of heterogeneous treatment effects (HTE) are common in applied causal inference research. However, when outcomes are latent variables assessed via psychometric instruments such as educational tests, standard methods ignore the potential HTE that may exist among the individual items of the outcome measure. Failing to account for “item-level” HTE (IL-HTE) can lead to both underestimated standard errors and identification challenges in the estimation of treatment-by-covariate interaction effects. We demonstrate how Item Response Theory (IRT) models that estimate a treatment effect for each assessment item can both address these challenges and provide new insights into HTE generally. This study articulates the theoretical rationale for the IL-HTE model and demonstrates its practical value using 75 datasets from 48 randomized controlled trials containing 5.8 million item responses in economics, education, and health research. Our results show that the IL-HTE model reveals item-level variation masked by single-number scores, provides more meaningful standard errors in many settings, allows for estimates of the generalizability of causal effects to untested items, resolves identification problems in the estimation of interaction effects, and provides estimates of standardized treatment effect sizes corrected for attenuation due to measurement error.","PeriodicalId":48105,"journal":{"name":"Journal of Policy Analysis and Management","volume":"16 1","pages":""},"PeriodicalIF":3.8,"publicationDate":"2025-06-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144329142","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Organizational adaptation, task complexity, and effective administration of unemployment programs in the American states","authors":"George A. Krause, Ji Hyeun Hong","doi":"10.1002/pam.70024","DOIUrl":"https://doi.org/10.1002/pam.70024","url":null,"abstract":"IT modernization reforms seek to improve administrative performance by improving the delivery of program benefits. Performance benefits manifest in a reduction in agency‐induced administrative errors, and a reduction in performance gaps between high and low complexity task caseloads. These claims are evaluated by assessing the impact of IT modernization reforms instituted by state unemployment insurance payment (UIP) agencies from 2002 to 2022. The evidence reveals that these reforms have discernible, unconditional dynamic effects, lowering overall program error rates by 20.95% over 60 months, as well as reducing both absolute and relative reduction in benefit overpayment error rates relating to program efficiency. IT reforms close the performance gap for overall program error rates between high and low task complexity caseloads involving individuals seeking different occupations. This evidence corroborates existing claims that technological‐based administration is inherently non‐neutral since program efficiency gains are emphasized relative to program accessibility gains.","PeriodicalId":48105,"journal":{"name":"Journal of Policy Analysis and Management","volume":"92 1","pages":""},"PeriodicalIF":3.8,"publicationDate":"2025-06-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144260678","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Elena Doty, Thomas J. Kane, Tyler Patterson, Douglas O. Staiger
{"title":"What do changes in state NAEP scores imply for birth cohorts’ later life outcomes?","authors":"Elena Doty, Thomas J. Kane, Tyler Patterson, Douglas O. Staiger","doi":"10.1002/pam.70018","DOIUrl":"https://doi.org/10.1002/pam.70018","url":null,"abstract":"Since 1990, the National Assessment of Educational Progress (NAEP) has been the primary benchmark for tracking the progress of state education reform. The focus on math and reading achievement is motivated by the cross‐sectional relationship between test scores and adult outcomes, such as earnings and college completion. But do changes in NAEP scores predict changes in long‐term economic and social outcomes for future earners—or do they reflect other factors unrelated to earnings such as teaching to the test? We investigate by linking long‐term outcomes by year and state of birth to NAEP scores. We find that more recent birth cohorts in states with large increases in NAEP math achievement enjoyed higher incomes, improved educational attainment, and declines in teen motherhood, incarceration, and arrest rates compared to those in states with smaller increases. In fact, the relationship between changes in NAEP achievement and cohort earnings is about two thirds the size of the cross‐sectional relationship observed in prior research: a 6% to 8% rise in earnings per standard deviation rise in 8th grade math. The results are not sensitive to controls for student demographics, labor market conditions, or measures of children's health (such as low birthweight).","PeriodicalId":48105,"journal":{"name":"Journal of Policy Analysis and Management","volume":"46 1","pages":""},"PeriodicalIF":3.8,"publicationDate":"2025-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144236968","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Credit card rewards and their costs","authors":"","doi":"10.1002/pam.70014","DOIUrl":"https://doi.org/10.1002/pam.70014","url":null,"abstract":"","PeriodicalId":48105,"journal":{"name":"Journal of Policy Analysis and Management","volume":"12 1","pages":""},"PeriodicalIF":3.8,"publicationDate":"2025-06-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144219288","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Who do credit cards reward?","authors":"Natasha Sarin","doi":"10.1002/pam.70022","DOIUrl":"https://doi.org/10.1002/pam.70022","url":null,"abstract":"","PeriodicalId":48105,"journal":{"name":"Journal of Policy Analysis and Management","volume":"38 1","pages":""},"PeriodicalIF":3.8,"publicationDate":"2025-06-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144218675","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Relaxing electoral constraints in local education funding","authors":"Michel Grosz, Ross Milton","doi":"10.1002/pam.70023","DOIUrl":"https://doi.org/10.1002/pam.70023","url":null,"abstract":"","PeriodicalId":48105,"journal":{"name":"Journal of Policy Analysis and Management","volume":"7 1","pages":""},"PeriodicalIF":3.8,"publicationDate":"2025-05-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144153461","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Pioneering Progress: American Science, Technology, and Innovation Policy by William B. Bonvillian. Cambridge, MA: MIT Press, 2024, 416 pp., $50.00 (paperback). ISBN: 978-0262549448.","authors":"Kai N. Lee","doi":"10.1002/pam.70017","DOIUrl":"https://doi.org/10.1002/pam.70017","url":null,"abstract":"Click on the article title to read more.","PeriodicalId":48105,"journal":{"name":"Journal of Policy Analysis and Management","volume":"133 1","pages":""},"PeriodicalIF":3.8,"publicationDate":"2025-05-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144133658","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}