Evaluation Review最新文献

{"title":"An Empirical Validation of the Regression Point Displacement Design Using Within-Study Comparison Logic: Emerging Possibilities and Cautions.","authors":"Joshua Hendrickse,&nbsp;William H Yeaton","doi":"10.1177/0193841X211064420","DOIUrl":"https://doi.org/10.1177/0193841X211064420","url":null,"abstract":"<p><strong>Background: </strong>The regression point displacement (RPD) design is a quasi-experiment (QE) that aims to control many threats to internal validity. Though it has existed for several decades, RPD has only recently begun to answer applied research questions in lieu of stronger QEs.</p><p><strong>Objectives: </strong>Our primary objective was to implement within-study comparison (WSC) logic to create RPD replicates and to determine conditions under which RPD might provide estimates comparable to those found in validating experiments.</p><p><strong>Research design: </strong>We utilize three randomized controlled trials (two cluster-level, one individual-level), artificially decomposing or creating cluster structures, to create multiple RPDs. We compare results in each RPD treatment group to a fixed set of control groups to gauge the congruence of these repeated RPD realizations with results found in these three RCTs.</p><p><strong>Results: </strong>RPD's performance was uneven. Using multiple criteria, we found that RPDs successfully predicted the direction of the RCT's intervention effect but inconsistently fell within the .10 SD threshold. A scant 13% of RPD results were statistically significant at either the .05 or .01 alpha-level. RPD results were within the 95% confidence interval of RCTs around half the time, and false negative rates were substantially higher than false positive rates.</p><p><strong>Conclusions: </strong>RPD consistently underestimates treatment effects in validating RCTs. We analyze reasons for this insensitivity and offer practical suggestions to improve the chances RPD will correctly identify favorable results. We note that the synthetic, \"decomposition of cluster RCTs,\" WSC design represents a prototype for evaluating other QEs.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"45 6","pages":"279-308"},"PeriodicalIF":0.9,"publicationDate":"2021-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"39870945","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Promoting Readiness of Minors in Supplemental Security Income (PROMISE): Early Impacts from a Multi-Site Random Assignment Evaluation.","authors":"Ankita Patnaik,&nbsp;Michael Levere,&nbsp;Gina Livermore,&nbsp;Arif Mamun,&nbsp;Jeffrey Hemmeter","doi":"10.1177/0193841X211055588","DOIUrl":"https://doi.org/10.1177/0193841X211055588","url":null,"abstract":"<p><strong>Background: </strong>PROMISE was a federal initiative to support youth receiving Supplemental Security Income (SSI) during the transition to adulthood.</p><p><strong>Objectives: </strong>This article presents estimates of the impacts of the six PROMISE projects on youth and family outcomes as of 18 months after enrolling in PROMISE.</p><p><strong>Research design: </strong>The study uses a randomized controlled trial design.</p><p><strong>Subjects: </strong>The six PROMISE projects each enrolled a minimum of 2000 treatment and control youth (and their parents) residing in their service areas who were aged 14 to 16 and receiving SSI.</p><p><strong>Measures: </strong>We estimated impacts on outcomes related to youth and family service use, school enrollment, training, employment, earnings, and federal disability program participation using survey and administrative data.</p><p><strong>Results: </strong>The projects succeeded in connecting more youth to transition services and more families to support services during the 18 months after enrollment, and most increased the likelihood that youth applied for state vocational rehabilitation services. On average, there was no impact on youth's school enrollment, but there were favorable impacts on youth's receipt of job-related training, employment, earnings, and total income. The projects did not affect parents' employment, earnings, or income, on average. For most outcomes PROMISE affected, the impacts varied substantially across the projects.</p><p><strong>Conclusions: </strong>The positive short-term impacts of PROMISE on youth's use of transition services, youth employment, and families' use of services are consistent with the program logic model and suggest there might be potential for longer-term favorable impacts on youth and family outcomes.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"45 5","pages":"228-270"},"PeriodicalIF":0.9,"publicationDate":"2021-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"39905495","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Comments on Randall Brown's Paper, \"The Evolving Art of Program Evaluation,\" on the Occasion of Brown Receiving the Peter Rossi Award From APPAM, November 12, 2020.","authors":"Katherine Swartz","doi":"10.1177/0193841X211055939","DOIUrl":"https://doi.org/10.1177/0193841X211055939","url":null,"abstract":"At a 1986 conference about the Income Maintenance Experiments, Robert Solow commented, “Suppose society wants to do something that is the right thing to do...There will sometimes be economical doubts, worries that doing the right thing might be unsuspectedly costly. Awell-designed experiment can help find out, and the prevalence of weak results is not an obstacle” (Solow, 1986, p. 222). Randall Brown instilled that point in the various evaluations he led. As a result, his contributions to evaluation research have had significant impacts on policies affecting the 40% of Americans covered by Medicare or","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"45 5","pages":"271-275"},"PeriodicalIF":0.9,"publicationDate":"2021-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"39583411","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The Role of Sample Size to Attain Statistically Comparable Groups - A Required Data Preprocessing Step to Estimate Causal Effects With Observational Data.","authors":"Ana Kolar,&nbsp;Peter M Steiner","doi":"10.1177/0193841X211053937","DOIUrl":"https://doi.org/10.1177/0193841X211053937","url":null,"abstract":"<p><p><b>Background:</b> Propensity score methods provide data preprocessing tools to remove selection bias and attain statistically comparable groups - the first requirement when attempting to estimate causal effects with observational data. Although guidelines exist on how to remove selection bias when groups in comparison are large, not much is known on how to proceed when one of the groups in comparison, for example, a treated group, is particularly small, or when the study also includes lots of observed covariates (relative to the treated group's sample size). <b>Objectives:</b> This article investigates whether propensity score methods can help us to remove selection bias in studies with small treated groups and large amount of observed covariates. <b>Measures:</b> We perform a series of simulation studies to study factors such as sample size ratio of control to treated units, number of observed covariates and initial imbalances in observed covariates between the groups of units in comparison, that is, selection bias. <b>Results:</b> The results demonstrate that selection bias can be removed with small treated samples, but under different conditions than in studies with large treated samples. For example, a study design with 10 observed covariates and eight treated units will require the control group to be at least 10 times larger than the treated group, whereas a study with 500 treated units will require at least, only, two times bigger control group. <b>Conclusions:</b> To confirm the usefulness of simulation study results for practice, we carry out an empirical evaluation with real data. The study provides insights for practice and directions for future research.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"45 5","pages":"195-227"},"PeriodicalIF":0.9,"publicationDate":"2021-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"39558482","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The Effectiveness of a Formal Financial Education Program at Primary Schools and the Role of Informal Financial Education.","authors":"Flavia Coda Moscarola,&nbsp;Adriaan Kalwij","doi":"10.1177/0193841X211042515","DOIUrl":"https://doi.org/10.1177/0193841X211042515","url":null,"abstract":"<p><p><b>Objectives:</b> This study examines the effectiveness of a formal financial education program for improving the financial literacy of primary school children and how this effectiveness is influenced by informal financial education provided by parents, such as giving pocket money and discussing money matters. <b>Method:</b> A quasi field experiment was carried out at the Museum of Saving in Turin where children participated in a financial education program (the treatment). The first two out of three classes that arrived at the museum were assigned to the treatment group and the third one to the comparison group. Difference-in-differences models are estimated using financial literacy data from a pretest taken about 1 week before the visit to the museum and a posttest taken on the day of the visit; just before starting with the program at the museum for the comparison group and just after program completion for the treatment group. <b>Results:</b> In line with previous studies, we find that our formal financial education program had a positive effect on the financial literacy of primary school children. The empirical findings provide weak evidence that this effect of formal financial education is stronger for children who received informal financial education from their parents. <b>Conclusions:</b> Our study contributes to the previous literature by presenting further evidence that a short extra-curricular course can be effective in increasing economic and financial literacy among students. Furthermore, we present suggestive evidence-worth of further research-that informal financial education can reinforce the effect of formal financial education.JEL Codes: A29, C93, G40.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"45 3-4","pages":"107-133"},"PeriodicalIF":0.9,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"39492675","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Are California's Local Flavored Tobacco Sales Restrictions Effective in Reducing the Retail Availability of Flavored Tobacco Products? A Multicomponent Evaluation.","authors":"Elizabeth Andersen-Rodgers,&nbsp;Xueying Zhang,&nbsp;Tam D Vuong,&nbsp;Liz Hendrix,&nbsp;Cheryl Edora,&nbsp;Rebecca J Williams,&nbsp;Lauren Groves,&nbsp;April Roeseler,&nbsp;Todd Rogers,&nbsp;David H Voelker,&nbsp;Nina C Schleicher,&nbsp;Trent O Johnson,&nbsp;Lisa Henriksen","doi":"10.1177/0193841X211051873","DOIUrl":"https://doi.org/10.1177/0193841X211051873","url":null,"abstract":"<p><strong>Introduction: </strong>Flavored tobacco appeals to new users. This paper describes evaluation results of California's early ordinances restricting flavored tobacco sales.</p><p><strong>Methods: </strong>A multicomponent evaluation of proximal policy outcomes involved the following: (a) tracking the reach of local ordinances; (b) a retail observation survey; and (c) a statewide opinion poll of tobacco retailers. Change in the population covered by local ordinances was computed. Retail observations compared availability of flavored tobacco at retailers in jurisdictions with and without an ordinance. Mixed models compared ordinance and matched no-ordinance jurisdictions and adjusted for store type. An opinion poll assessed retailers' awareness and ease of compliance with local ordinances, comparing respondents in ordinance jurisdictions with the rest of California.</p><p><strong>Results: </strong>The proportion of Californians living in a jurisdiction with an ordinance increased from 0.6% in April 2015 to 5.82% by January 1, 2019. Flavored tobacco availability was significantly lower in ordinance jurisdictions than in matched jurisdictions: menthol cigarettes (40.6% vs. 95.0%), cigarillos/cigar wraps with explicit flavor descriptors (56.4% vs. 85.0%), and vaping products with explicit flavor descriptors (6.1% vs. 56.9%). Over half of retailers felt compliance was easy; however, retailers in ordinance jurisdictions expressed lower support for flavor sales restrictions.</p><p><strong>Conclusions: </strong>The proportion of California's population covered by a flavor ordinance increased nine-fold between April 2015 and January 2019. Fewer retailers in ordinance jurisdictions had flavored tobacco products available compared to matched jurisdictions without an ordinance, but many still advertised flavored products they could not sell. Comprehensive ordinances and retailer outreach may facilitate sales-restriction support and compliance.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"45 3-4","pages":"134-165"},"PeriodicalIF":0.9,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/49/ca/10.1177_0193841X211051873.PMC8600589.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"39556071","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Stakeholder Involvement in Evaluation: Does it Affect Observers' Perceptions of an Evaluation? And Which Stakeholder Group(s) Do People Think Should to Participate?","authors":"Melvin M Mark,&nbsp;Julian B Allen,&nbsp;Joshuah L Goodwin","doi":"10.1177/0193841X211055937","DOIUrl":"https://doi.org/10.1177/0193841X211055937","url":null,"abstract":"<p><strong>Background: </strong>Stakeholders are often involved in evaluation, such as in the selection of specific research questions and the interpretation of results. Except for the topic of whether stakeholder involvement increases use, a paucity of research exists to guide practice regarding stakeholders.</p><p><strong>Objectives: </strong>We address two questions: (1) If a third-party observer knows stakeholders were involved in an evaluation, does that affect the perceived credibility, fairness, and relevance of the evaluation? (2) Among individuals with a possible stake in an evaluation, which stakeholder group(s) do they want to see participate; in particular, do they prefer that multiple stakeholder groups, rather than a single group, participate?</p><p><strong>Research design: </strong>Six studies are reported. All studies address the former question, while Studies 3 to 5 also focus on the latter question. To study effects of stakeholder involvement on third-party views, participants read summaries of ostensible evaluations, with stakeholder involvement noted or not. To examine a priori preferences among potential stakeholders, participants completed a survey about alternative stakeholder group involvement in an evaluation in which they would likely have an interest.</p><p><strong>Results and conclusions: </strong>Across studies, effects of reported stakeholder participation on third-parties' views were not robust; however, small effects on perceived fairness sometimes, but not always, occurred after stakeholder involvement and its rationales had been made salient. All surveys showed a large preference for the involvement of multiple, rather than single stakeholder groups. We discuss implications for research and practice regarding stakeholder involvement, and for research on evaluation more generally.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"45 3-4","pages":"166-190"},"PeriodicalIF":0.9,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"39557684","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Neural Networks to Estimate Generalized Propensity Scores for Continuous Treatment Doses.","authors":"Zachary K Collier, Walter L Leite, Allison Karpyn","doi":"10.1177/0193841X21992199","DOIUrl":"10.1177/0193841X21992199","url":null,"abstract":"<p><strong>Background: </strong>The generalized propensity score (GPS) addresses selection bias due to observed confounding variables and provides a means to demonstrate causality of continuous treatment doses with propensity score analyses. Estimating the GPS with parametric models obliges researchers to meet improbable conditions such as correct model specification, normal distribution of variables, and large sample sizes.</p><p><strong>Objectives: </strong>The purpose of this Monte Carlo simulation study is to examine the performance of neural networks as compared to full factorial regression models to estimate GPS in the presence of Gaussian and skewed treatment doses and small to moderate sample sizes.</p><p><strong>Research design: </strong>A detailed conceptual introduction of neural networks is provided, as well as an illustration of selection of hyperparameters to estimate GPS. An example from public health and nutrition literature uses residential distance as a treatment variable to illustrate how neural networks can be used in a propensity score analysis to estimate a dose-response function of grocery spending behaviors.</p><p><strong>Results: </strong>We found substantially higher correlations and lower mean squared error values after comparing true GPS with the scores estimated by neural networks. The implication is that more selection bias was removed using GPS estimated with neural networks than using GPS estimated with classical regression.</p><p><strong>Conclusions: </strong>This study proposes a new methodological procedure, neural networks, to estimate GPS. Neural networks are not sensitive to the assumptions of linear regression and other parametric models and have been shown to be a contender against parametric approaches to estimate propensity scores for continuous treatments.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"193841X21992199"},"PeriodicalIF":0.9,"publicationDate":"2021-03-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9344588/pdf/nihms-1824103.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"25422653","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Imputation of Missing Covariate Data Prior to Propensity Score Analysis: A Tutorial and Evaluation of the Robustness of Practical Approaches","authors":"Walter L. Leite, Burak Aydin, Dee D. Cetin-Berber","doi":"10.1177/0193841X211020245","DOIUrl":"https://doi.org/10.1177/0193841X211020245","url":null,"abstract":"Background: Propensity score analysis (PSA) is a popular method to remove selection bias due to covariates in quasi-experimental designs, but it requires handling of missing data on covariates before propensity scores are estimated. Multiple imputation (MI) and single imputation (SI) are approaches to handle missing data in PSA. Objectives: The objectives of this study are to review MI-within, MI-across, and SI approaches to handle missing data on covariates prior to PSA, investigate the robustness of MI-across and SI with a Monte Carlo simulation study, and demonstrate the analysis of missing data and PSA with a step-by-step illustrative example. Research design: The Monte Carlo simulation study compared strategies to impute missing data in continuous and categorical covariates for estimation of propensity scores. Manipulated conditions included sample size, the number of covariates, the size of the treatment effect, missing data mechanism, and percentage of missing data. Imputation strategies included MI-across and SI by joint modeling or multivariate imputation by chained equations (MICE). Results: The results indicated that the MI-across method performed well, and SI also performed adequately with smaller percentages of missing data. The illustrative example demonstrated MI and SI, propensity score estimation, calculation of propensity score weights, covariate balance evaluation, estimation of the average treatment effect on the treated, and sensitivity analysis using data from the National Longitudinal Survey of Youth.","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"45 1","pages":"34 - 69"},"PeriodicalIF":0.9,"publicationDate":"2021-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/0193841X211020245","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46316974","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Comparing Performance of Methods to Deal With Differential Attrition in Randomized Experimental Evaluations.","authors":"Kaitlin Anderson,&nbsp;Gema Zamarro,&nbsp;Jennifer Steele,&nbsp;Trey Miller","doi":"10.1177/0193841X211034363","DOIUrl":"https://doi.org/10.1177/0193841X211034363","url":null,"abstract":"<p><p><b>Background:</b> In randomized controlled trials, attrition rates often differ by treatment status, jeopardizing causal inference. Inverse probability weighting methods and estimation of treatment effect bounds have been used to adjust for this bias. <b>Objectives:</b> We compare the performance of various methods within two samples, both generated through lottery-based randomization: one with considerable differential attrition and an augmented dataset with less problematic attrition. <b>Research Design:</b> We assess the performance of various correction methods within the dataset with problematic attrition. In addition, we conduct simulation analyses. <b>Results:</b> Within the more problematic dataset, we find the correction methods often performed poorly. Simulation analyses indicate that deviations from the underlying assumptions for bounding approaches damage the performance of estimated bounds. <b>Conclusions:</b> We recommend the verification of the underlying assumptions in attrition correction methods whenever possible and, when verification is not possible, using these methods with caution.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"45 1-2","pages":"70-104"},"PeriodicalIF":0.9,"publicationDate":"2021-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"39298023","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}