Mariangel Pacheco-Troisi, Mónica García-Melón, Fernando Jiménez-Sáez
{"title":"Anticipatory evaluation. How to incorporate an anticipatory technique into a theory-driven evaluation process. Results of application in a case study.","authors":"Mariangel Pacheco-Troisi, Mónica García-Melón, Fernando Jiménez-Sáez","doi":"10.1016/j.evalprogplan.2024.102509","DOIUrl":"https://doi.org/10.1016/j.evalprogplan.2024.102509","url":null,"abstract":"<p><p>In recent years, there has been increased focus on strategic learning from impact analysis, including in the field of science, technology, and innovation. In this paper, we propose combining techniques from two fields of study. Firstly, we adopt the approach of addressing impact through productive interactions between science and society, and secondly, we incorporate an anticipatory dimension by integrating game analysis involving key actors. Through a theory-driven evaluation design, we consider expected impacts as promises of the future. Within an anticipatory perspective, the future can be shaped by the interactions among different actors in the present. In this article, we apply this approach step-by-step to a research institute program in Uruguay. We demonstrate how the achieved results offer strategic insights to the program manager for anticipating and attaining the desired impacts. Additionally, we provide summative inputs for accountability using a flexible technique applicable at any stage of the program life cycle. The article concludes with a discussion of the advantages and disadvantages compared to other techniques, along with lessons learned that may benefit other evaluators seeking to replicate this approach. Furthermore, we explore potential extensions and opportunities for further improvement in this research.</p>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2024-10-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142477833","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"What evaluation criteria are used in policy evaluation research: A cross-field literature review","authors":"","doi":"10.1016/j.evalprogplan.2024.102512","DOIUrl":"10.1016/j.evalprogplan.2024.102512","url":null,"abstract":"<div><div>This literature review offers a comprehensive overview of the use of evaluation criteria across five policy fields: social services, land-use planning, teaching in higher education, vocational education, and the environment. Though it is a key part of the evaluation process, the question of how criteria are defined, chosen, and applied generates surprisingly little debate among the evaluation community. In evaluation practice, criteria are often taken for granted – and occasionally even used in ways that are neither explicit nor transparent. This cross-field literature review shows a strong presence of routinized evaluation criteria (relating to the specifics of each policy field), while some new sets of higher-degree criteria also emerge in the face of social challenges relating to sustainability, public acceptance, or social justice. Criteria development draws on both inductive bottom-up processes (which can include policy stakeholders) and top-down deductive processes (which derive criteria from the literature, as well as from national and international standards). A more profound reflection on evaluation criteria (that is, the dimensions used by societies to assess the success of policy interventions) might be required in the future of evaluation research and planning; a deeper cross-field dialogue could support this endeavor.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2024-10-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142444898","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A program evaluation of the new choices workforce development program: An appreciative inquiry approach","authors":"","doi":"10.1016/j.evalprogplan.2024.102507","DOIUrl":"10.1016/j.evalprogplan.2024.102507","url":null,"abstract":"<div><div>A significant amount of money ($1.76B annually in the United States) is spent on workforce development programs, while there is limited research on the effectiveness of workforce development programs in meeting their program objectives and assisting program participants in attaining employment. This study evaluated the New Choices Program, a workforce development program offered by PA Women Work, to help its clients obtain employment and overcome personal and professional barriers. The program has historically been offered in a 30-hour in person format but was forced to be modified to a 10-hour virtual program when the COVID-19 pandemic restrictions began. This program evaluation included a comparative analysis of the perceptions of participants in the 30-hour in person program and the 10-hour virtual program. It was found that participants in both the 30-hour in person program and 10-hour virtual program perceived the program positively, experienced an increase in self-confidence and belonging, which led to either obtaining employment or being better prepared for the job search process. The data will help inform the New Choices program stakeholders on programmatic improvements and how best to structure the program in the post-pandemic employment world.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2024-10-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142438413","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"The Problems (and possible solutions) of assessing risk, race and recidivism in long operating drug treatment courts","authors":"","doi":"10.1016/j.evalprogplan.2024.102510","DOIUrl":"10.1016/j.evalprogplan.2024.102510","url":null,"abstract":"<div><div>Formal criminogenic risk tools can be an important control in assessing racial inequities in access to treatment courts and in evaluating both proximal and distal outcomes from those programs. To achieve this potential, however, it is important that risk tools themselves operate in a racially neutral fashion and that they operate consistently over the period assessed. Tools that are not properly calibrated by race and changes in the tools used over the life of a program are therefore significant evaluation concerns. Our paper is the first to assess the adequacy of an important risk-needs instrument, the LSI-R, across racial groups in a drug treatment court setting. The main contribution of the current study is not as a test of that instrument, which has been widely studied in other settings. Rather, because two different criminogenic risk tools were used over the study time period, we took this opportunity to explore the use of a readily constructible “proxy” measure of risk to support analysis of risk and race interactions over the life of the program.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2024-10-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142433558","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"The criticality of language: Exploring STEM education evaluators conceptualizations of equity, diversity, and inclusion and the influence on their roles and practice","authors":"","doi":"10.1016/j.evalprogplan.2024.102511","DOIUrl":"10.1016/j.evalprogplan.2024.102511","url":null,"abstract":"<div><div>Language is critical in evaluation. It influences understanding of goal attainment, judgments made about the quality of a program, interactions between stakeholder groups, interpretation of findings, and future actions. This paper reports research on language that unearths complexities related to the agenda of diversifying STEM and the utilization of evaluation in achieving its goal through an examination of STEM education evaluators’ conceptualizations of diversity, equity, and inclusion and the implications for their practice. Data was collected via semi-structured interviews. Findings revealed (1) the multifaceted nature of terms such as equity, diversity, and inclusion, (2) the role of an evaluator is not singular, and (3) the importance of surfacing the ways in which language is both value-laden and socially shaping for potentially disrupting one’s knowledge of status quo injustices.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2024-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142394298","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"“What is our actual impact?”: A mixed-method assessment of a Belgian shelter for homeless men","authors":"","doi":"10.1016/j.evalprogplan.2024.102508","DOIUrl":"10.1016/j.evalprogplan.2024.102508","url":null,"abstract":"<div><div>This paper presents an outcomes assessment carried out in an 84-bed long-term shelter for homeless men in Brussels. In the context of increasing Housing First studies, it investigates an instance of the traditional “treatment first” model and provides a new Belgian case study. Adopting a multidimensional approach, it aims to better understand what impact an average stay in a long-term shelter has on its residents. The assessment investigated several outcomes—income and housing, physical and mental health, life skills, social and assistance network—and relied on a participative mixed-method design. Although the shelter mission is broad and ambitious (i.e. autonomy, global well-being and reintegration into society), the assessment results show that the shelter struggles to have positive effects on the residents beyond the provision of basic care (a roof, food, administrative support) and that the stress felt by the residents even tends to increase during their stay. Several recommendations collectively emerged from the assessment: individualizing shelter support and making it evolve during the stay, reducing the size of the shelter while at the same time fostering community living, developing partnerships. At the public policy level, we would recommend revising the mission of long-term shelters in accordance with their means.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2024-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142420394","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A theoretical framework to companies value creation through a systematic review of intangibles’ management","authors":"","doi":"10.1016/j.evalprogplan.2024.102506","DOIUrl":"10.1016/j.evalprogplan.2024.102506","url":null,"abstract":"<div><div>The purpose of this article is to develop a theoretical framework that identifies the intangible aspects that can be managed and contribute to the value creation for organizations. The theoretical framework was developed based on a systematic review performed according to a protocol that proposes steps to identify the intangible aspects present in the scientific literature. Mendeley software assisted in organizing and reading the 3152 articles identified by the systematic review. The results of the article propose a classification of intangible aspects identified in levels that collaborate with the value creation in organizations. The first and main group being called FPVs, encompassing: reputation, innovation, performance, legitimacy, and knowledge. Strategically, the FPVs are subdivided into 15 CSFs, that cover the other 35 intangibles perceived in the studies, called Indicators and that can be managed. The results of the article provide theoretical and managerial implications and can be used by the academic community and by managers of industrial organizations. The results present reflections on how intangible aspects are present in research and the management of organizations. Furthermore, the literature review proves the importance of analyzing and monitoring intangible aspects nowadays.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2024-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142394297","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Modeling a needs assessment approach for policymakers to investigate, understand, and reduce gun violence","authors":"","doi":"10.1016/j.evalprogplan.2024.102505","DOIUrl":"10.1016/j.evalprogplan.2024.102505","url":null,"abstract":"<div><div>We conducted a needs assessment evaluation as part of a community effort to plan tailored responses to reduce gun violence in a midwestern town. Various data sources were used to analyze factors leading to gun violence. The results guided the creation of evidence-based response strategies. Study findings demonstrate that even in a small town, the nature and drivers of gun violence vary considerably at the micro level and thus should be examined at this level. In this paper, we present the recommendations that followed from our study, not as solutions for other jurisdictions to adopt on their face, but rather as an example of 1) the types of solutions that follow from a thorough analysis of a local problem, and 2) a plethora of potential responses by social institutions. The goal of the paper is to provide an example process for policymakers from jurisdictions of various sizes and locales of the application of a problem-oriented approach to understanding and preventing gun violence. Undertaking a comprehensive, micro-focus to analyze the gun violence problem and its drivers in a jurisdiction provides essential information to guide the formulation of tailored, evidence-based responses.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2024-09-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142337015","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"To scope or not to scope? The benefits and challenges of integrating scoping studies in rapid qualitative research and evaluation","authors":"","doi":"10.1016/j.evalprogplan.2024.102495","DOIUrl":"10.1016/j.evalprogplan.2024.102495","url":null,"abstract":"<div><p>Creating sustainable change and fostering collaborative relationships between researchers and stakeholders is a recognized challenge in the field of evaluation. Identifying programme purpose, cultural context, potential challenges, and engaging stakeholders before an evaluation can produce responsive and impactful evaluations. This paper discusses implementing a targeted scoping study within the framework of rapid qualitative research and evaluation. A scoping study enables collaborative decision-making on evaluation priorities, and functions as an evaluability assessment in time-sensitive contexts. In our experience, a scoping study can be carried out in as little as five days or as long as six weeks. It is timely to revisit the question of what factors influence evaluation outcomes, a scoping study can be used to support an evaluation, address access to data and research, and strengthen communication channels. The methodological approach was used to co-produce an evaluation with an NGO that accurately reflected their needs, recognizing possible challenges and solutions.</p></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2024-09-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0149718924000971/pdfft?md5=e7b517fda6e66e5253068b449d13b057&pid=1-s2.0-S0149718924000971-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142172993","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Arts-based evaluation of the Communities ChooseWell program","authors":"","doi":"10.1016/j.evalprogplan.2024.102496","DOIUrl":"10.1016/j.evalprogplan.2024.102496","url":null,"abstract":"<div><p>Arts-based evaluation is an effective and fun way to engage people and uncover meaningful, valid results. In this project, an arts-based approach was used to gain an understanding of the effects the Communities ChooseWell Program has had according to the Champions’ experiences. We wanted to identify what changes, if any, has Communities ChooseWell fostered through the past 10 years? This evaluation was completed using an arts-based approach which allowed us to explore varied long-term effects in different contexts. The creative process allowed for an open approach not predetermining the nature of potential effects. It also gave the participants space to identify what matters the most according to the ChooseWell Champions. This evaluation was in addition to evaluation requirements from the ChooseWell programs funder. In this article we will first present the context of evaluation and state our positionality. We will then present the methodology and methods. Finally, we will present and discuss the results and recommendations.</p></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2024-09-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S0149718924000983/pdfft?md5=6c1c147bb27c304dd34c36a307a75f54&pid=1-s2.0-S0149718924000983-main.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142172994","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}