Evaluation ReviewPub Date : 2025-10-01Epub Date: 2025-03-24DOI: 10.1177/0193841X251329459
Qiqi Cheng, Neil Humphrey
{"title":"Preliminary Evaluation of a Targeted, School-Based Social and Emotional Learning Intervention for at Risk Youth: Football Beyond Borders.","authors":"Qiqi Cheng, Neil Humphrey","doi":"10.1177/0193841X251329459","DOIUrl":"10.1177/0193841X251329459","url":null,"abstract":"<p><p>This brief report provides preliminary independent evidence of the efficacy of Football Beyond Borders (FBB), a targeted, school-based social and emotional learning (SEL) intervention for at-risk youth. FBB includes weekly SEL classroom sessions, activities on the football pitch, 1:1 therapy sessions, holiday support, and rewards trips. Propensity score matching and difference-in-differences estimation were used in a pre-test/post-test control group design to assess the impact of FBB on the mental wellbeing (assessed via the Short Warwick-Edinburgh Mental Wellbeing Scale, SWEMWBS) of participants designated at-risk (<i>N</i> = 46 aged 12-14, 78.3% male), passive learners (<i>N</i> = 72, aged 12-14, 84.7% male), and role models (<i>N</i> = 35, aged 12-14, 85.7% male), with matched control samples derived from a subset of the #BeeWell cohort study (<i>N</i> = 8015). A significant intervention effect was observed for at-risk youth, with FBB leading to an increase of approximately 2.4 SWEMWBS points (<i>d</i> = 0.44). No significant intervention effects were observed for passive learners or role models. These results indicate that FBB can improve the mental wellbeing of at-risk youth. Accordingly, an explanatory trial is warranted.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"914-930"},"PeriodicalIF":3.7,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12379023/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143694098","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Evaluation ReviewPub Date : 2025-10-01Epub Date: 2025-01-11DOI: 10.1177/0193841X251314731
Hugh Sharma Waddington, Hikari Umezawa, Howard White
{"title":"What Can We Learn From Qualitative Impact Evaluations About the Effectiveness of Lobby and Advocacy? A Meta-Evaluation of Dutch aid Programmes and Assessment Tool.","authors":"Hugh Sharma Waddington, Hikari Umezawa, Howard White","doi":"10.1177/0193841X251314731","DOIUrl":"10.1177/0193841X251314731","url":null,"abstract":"<p><p>Official development agencies are increasingly supporting civil society lobby and advocacy (L&A) to address poverty and human rights. However, there are challenges in evaluating L&A. As programme objectives are often to change policies or practices in a single institution like a Government Ministry, L&A programmes are often not amenable to large-n impact evaluation methods. They often work in strategic partnerships to foster change; hence, contribution may be a more relevant evaluation question than attribution. Small-n qualitative approaches are available to measure the effectiveness of L&A which use the theory of change as their analytical framework. We conducted a meta-evaluation of 36 evaluations of multi-component international programmes to support civil society L&A across Asia, Africa and Latin America, comprising the majority of programmatic support from one international donor. We assessed the confidence in causal claims in the evaluations using a new tool that we developed. Assessments of the contribution of the programmes to the changes in outcomes were not provided in many of the evaluations, nor were predictable sources of bias addressed. Given that L&A programmes are likely to adopt an influencing approach where many different inside-track and outside-track engagement objectives, opportunities and strategies are attempted, many of which might be expected to fail, there appeared to be a clear bias in the evaluations towards reporting outcomes that were achieved, ignoring those that were not. We provide guidance on how to improve the design, conduct and reporting of small-n qualitative evaluations of aid effectiveness.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"851-879"},"PeriodicalIF":3.7,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12379041/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142966781","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Evaluation ReviewPub Date : 2025-10-01Epub Date: 2025-04-11DOI: 10.1177/0193841X251328710
Shuangpeng Yang, Li Zhang
{"title":"Optimizing an Omnichannel Retail Strategy Considering Customer Segmentation.","authors":"Shuangpeng Yang, Li Zhang","doi":"10.1177/0193841X251328710","DOIUrl":"10.1177/0193841X251328710","url":null,"abstract":"<p><p>Unlike previous studies on fixed logistics nodes, this research explored how consumer distribution impacts store selection and inventory balance, integrating the <i>ship-from-store</i> strategy to increase fulfillment within multiperiod sales plans. Specifically, omnichannel retailers (O-tailer) must sequentially decide on inventory replenishment from suppliers to the distribution center (DC), allocation from the DC to stores, and which department will fulfill online orders. We introduce a multiperiod stochastic optimization model and solve it with a robust two-stage approach (RTA). In Stage 1, we use the K-means algorithm and silhouette coefficients to determine the optimal number of stores. In Stage 2, linear decision rule (LDR) are employed to decide on replenishment, allocation, and order fulfillment quantities. Numerical experiments show that RTA outperforms existing methods, achieving solutions with efficiency gaps of less than 10%, even when assumptions are not fully met. Additionally, the sensitivity analysis shows that variations in product prices, fulfillment costs, market share, and customer distribution consistently lead to greater profits with the <i>ship-from-store</i> strategy.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"814-850"},"PeriodicalIF":3.7,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143991817","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Evaluation ReviewPub Date : 2025-10-01Epub Date: 2025-03-12DOI: 10.1177/0193841X251326738
Thomas S Dee
{"title":"The Case for Preregistering Quasi-Experimental Program and Policy Evaluations.","authors":"Thomas S Dee","doi":"10.1177/0193841X251326738","DOIUrl":"10.1177/0193841X251326738","url":null,"abstract":"<p><p>The recognition that researcher discretion coupled with unconscious biases and motivated reasoning sometimes leads to false findings (\"p-hacking\") led to the broad embrace of study preregistration and other open-science practices in experimental research. Paradoxically, the preregistration of quasi-experimental studies remains uncommon although such studies involve far more discretionary decisions and are the most prevalent approach to making causal claims in the social sciences. I discuss several forms of recent empirical evidence indicating that questionable research practices contribute to the comparative unreliability of quasi-experimental research and advocate for adopting the preregistration of such studies. The implementation of this recommendation would benefit from further consideration of key design details (e.g., how to balance data cleaning with credible preregistration) and a shift in research norms to allow for appropriately nuanced sensemaking across prespecified, confirmatory results and other exploratory findings.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"931-945"},"PeriodicalIF":3.7,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143605535","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Evaluation ReviewPub Date : 2025-09-23DOI: 10.1177/0193841X251380908
Anne Revillard
{"title":"External Validity and Generalizability in Program Evaluation: Embracing Complexity.","authors":"Anne Revillard","doi":"10.1177/0193841X251380908","DOIUrl":"https://doi.org/10.1177/0193841X251380908","url":null,"abstract":"<p><p>This introduction to the second special issue of <i>Evaluation review</i> on external validity and generalizability opens a dialogue between different ways to think about generalizability in program evaluation. It argues that generalizability in impact evaluation fundamentally is about inferring some form of <i>causality</i> at a level broader than the specific circumstances of the initial study or studies from which these inferences are drawn. The question, then, is about how one apprehends causality: in other words, <i>what</i> is being generalized and <i>how</i>? The first special issue mainly relied on a <i>counterfactual</i> conception of causality, embodied by experimental and quasi-experimental methods, that aimed at impact <i>measurement</i>. The articles in this volume, drawing on mixed methods, also mobilize <i>generative</i> and <i>configurational</i> causal inferences to provide further levers of generalizability, focusing on <i>how</i> the impact is produced. The introduction insists on the specific input of qualitative methods in this respect, as theorized by grounded theory.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"193841X251380908"},"PeriodicalIF":3.7,"publicationDate":"2025-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145132150","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Evaluation ReviewPub Date : 2025-09-22DOI: 10.1177/0193841X251380898
E C Hedberg, Larry V Hedges
{"title":"Computing Statistical Power for the Difference in Differences Design.","authors":"E C Hedberg, Larry V Hedges","doi":"10.1177/0193841X251380898","DOIUrl":"https://doi.org/10.1177/0193841X251380898","url":null,"abstract":"<p><p>The difference in differences design is widely used to assess treatment effects in natural experiments or other situations where random assignment cannot, or is not, used (see, e.g., Angrist & Pischke, 2009). The researcher must make important decisions about which comparisons to make, the measurements to make, and perhaps the number of individuals whose data is included in each timepoint. Also, interpretation of any statistical results, particularly null results, is improved by understanding the sensitivity of the design. This paper describes methods for computing the statistical power for tests of treatment effects in the difference in differences design. We describe alternative approaches to the analysis of the design, show which are equivalent, and provide expressions for computing statistical power and determining minimum detectable effect sizes. We then discuss how these methods could be generalized to unbalanced designs, designs with covariates, and designs more than two timepoints including difference in difference in differences designs.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"193841X251380898"},"PeriodicalIF":3.7,"publicationDate":"2025-09-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145126111","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Evaluation ReviewPub Date : 2025-09-03DOI: 10.1177/0193841X251375503
Xiang Gao, Xiaolan Qiu, Beibei Shang
{"title":"Strategic Insights From Voluntary and Mandatory ESG Evaluations in a Competitive Market.","authors":"Xiang Gao, Xiaolan Qiu, Beibei Shang","doi":"10.1177/0193841X251375503","DOIUrl":"https://doi.org/10.1177/0193841X251375503","url":null,"abstract":"<p><p>Environmental, social, and governance (ESG) considerations are garnering increasing interest from various stakeholders. Nevertheless, the impact of ESG evaluation policies-whether voluntary or mandatory-are complex, particularly in competitive markets. Consequently, this study develops a game-theoretic model to investigate the propensity of competing manufacturers to invest in ESG initiatives. This study demonstrates that ESG evaluations reduce information asymmetry regarding ESG levels in competitive dynamics. However, this phenomenon may lower manufacturers' incentives to invest in ESG. Interestingly, this study finds that although mandatory ESG evaluations boost the overall profitability of competing manufacturers, they do not invariably lead to enhanced ESG outcomes.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"193841X251375503"},"PeriodicalIF":3.7,"publicationDate":"2025-09-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144993896","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Evaluation ReviewPub Date : 2025-08-28DOI: 10.1177/0193841X251370426
Thilo Bodenstein, Achim Kemmerling
{"title":"Taking Stock of Qualitative Methods of Evaluation: A Study of Practices and Quality Criteria.","authors":"Thilo Bodenstein, Achim Kemmerling","doi":"10.1177/0193841X251370426","DOIUrl":"https://doi.org/10.1177/0193841X251370426","url":null,"abstract":"<p><p>Research on evaluation has mapped the landscape of quantitative evaluation methods. There are far fewer overviews for qualitative methods of evaluation. We present a review of scholarly articles from five widely read evaluation research journals, examining the types of methods used and the transparency of their quality criteria. We briefly look at a large sample of 1070 articles and then randomly select 50 for in-depth study. We document a remarkable variety of qualitative methods, but some stand out: Case studies and stakeholder analysis, often combined with interview techniques. Articles rarely define and conceptualize their methods explicitly. This is understandable from a practical point of view, but it can make it difficult to critically interrogate findings and build systematic knowledge. Finally, we find that the transparency of qualitative criteria required in the literature is not always sufficient, which can hinder the synthesis of results.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"193841X251370426"},"PeriodicalIF":3.7,"publicationDate":"2025-08-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144974155","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Evaluation ReviewPub Date : 2025-08-20DOI: 10.1177/0193841X251369313
Dan Wang, Anh Ngoc Quang Huynh, Handong Tang
{"title":"Powering Innovation: How High-Performance Computing Platform Revolutionizes University Research.","authors":"Dan Wang, Anh Ngoc Quang Huynh, Handong Tang","doi":"10.1177/0193841X251369313","DOIUrl":"10.1177/0193841X251369313","url":null,"abstract":"<p><p>High-performance computing platforms (HPCP) serve as a critical component of institutional research infrastructure at universities, enhancing innovation efficiency. However, there is an absence of empirical research quantifying their impacts. Acknowledging this void, a difference-in-differences (DID) approach is employed in this study to systematically evaluate the influence of HPCP on university innovation efficiency. The findings reveal that: (1) universities with access to HPCP experience a significant improvement in innovation efficiency, with heterogeneities observed across regions, institutional rankings and university types; (2) HPCP drives innovation efficiency by attracting and cultivating top-tier researchers, expanding the scope of research activities and enabling the production of high-quality research outputs; and (3) the relationship between HPCP and innovation efficiency exhibits non-linearity, influenced by the quality and volume of research outputs as well as the presence of elite researchers. The contributions of this study are threefold viz: (1) providing empirical evidence quantifying HPCP's role in university innovation efficiency; (2) elucidating the mechanisms through which HPCP enhances innovation; and (3) identifying the non-linear relationship between HPCP and innovation efficiency, offering nuanced insights for policymakers to engender targeted strategies.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"193841X251369313"},"PeriodicalIF":3.7,"publicationDate":"2025-08-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144884105","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Evaluation ReviewPub Date : 2025-08-01Epub Date: 2024-12-23DOI: 10.1177/0193841X241305869
David C Seith, Siyanbade Adegoke, Camisha Burchett, Ryan Kennedy
{"title":"Time to Take a Chance: The Promise of Royston-Parmar Proportional Hazard Models for Understanding Caseload Transitions.","authors":"David C Seith, Siyanbade Adegoke, Camisha Burchett, Ryan Kennedy","doi":"10.1177/0193841X241305869","DOIUrl":"10.1177/0193841X241305869","url":null,"abstract":"<p><p>In this letter to the editor, we compare six different event history models to estimate <i>which</i> eligible families participated in a subsidized rental housing program and <i>when</i>. Answering these questions can inform efforts to improve program marketing and outreach, staffing and budgeting, triage, bias identification, as well as benchmarking and evaluation. One of six specifications clearly outperforms the others and understanding how will inform similar research pursuits. Although decision-relevant participation patterns are available in state administrative records, deciphering them is difficult for several well-known reasons. Participants enter and exit the eligible risk pool at different times, for different reasons, and at different rates. To answer our questions of <i>when</i> and <i>whom</i>, we restructure the data from calendar to relative months and then employ event history models designed to accurately estimate a complete hypothetical risk trajectory from observed spells of varying lengths, many of which ended before families took up the rental subsidy, (i.e., censored observation spells). We find that eligible parents most likely to take up the subsidy live in high-rent counties, have relatively strong recent work history, short prior adult lifetime TANF receipt, and medium-size families. Program take-up fell substantially during the COVID-19 pandemic. Contrasting the application of six parallel specifications, we find that a Royston-Parmar proportional hazard model achieves an exceptional balance between the descriptive accuracy of discrete time approaches and the elegance of Cox regression.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"773-796"},"PeriodicalIF":3.0,"publicationDate":"2025-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142877883","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}