Evaluation and Program Planning最新文献

{"title":"Established but unused? An empirical study on the impact of health records on the utilization of basic public health services in China","authors":"Xiafei Tao ,&nbsp;Si Shi ,&nbsp;Junqiang Liu","doi":"10.1016/j.evalprogplan.2025.102654","DOIUrl":"10.1016/j.evalprogplan.2025.102654","url":null,"abstract":"<div><h3>Objectives</h3><div>Health records are crucial for implementing basic public health services (BPHSs) in China, yet their utilization rate lags behind the coverage rate. This study aimed to provide insights into the underutilization issue by exploring whether health records facilitate the utilization of other BPHSs.</div></div><div><h3>Methods</h3><div>We conducted empirical analyses to examine the impact of health records on the utilization of two specific BPHSs: health education and health management services for patients with hypertension and type 2 diabetes mellitus (T2DM). Multiple linear regression and binary logistic models were employed in the analyses. Additionally, we used the propensity score matching approach to address potential biases.</div></div><div><h3>Results</h3><div>Health records significantly enhanced the diversity of health education attended by residents, fostered active engagement in health education, and facilitated the utilization of corresponding health management services for patients with hypertension and T2DM.</div></div><div><h3>Conclusions</h3><div>Despite widespread skepticism concerning the utilization rate of health records, our findings provided empirical evidence that health records may benefit citizens’ self-health management and the utilization of other BPHSs. However, the specific mechanism of this impact needs to be further investigated. This study offers insights into the evaluation of public health programs in the Chinese context and highlights their implications for promoting health equity.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":"112 ","pages":"Article 102654"},"PeriodicalIF":1.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144663488","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Evaluating sport-for-development outcome measures used in a living lab setting: Process, improvements, and insights","authors":"Bhanu Sharma ,&nbsp;Jackie Robinson ,&nbsp;Benjamin B. Arhen ,&nbsp;Brian W. Timmons ,&nbsp;Bryan Heal ,&nbsp;Marika Warner","doi":"10.1016/j.evalprogplan.2025.102647","DOIUrl":"10.1016/j.evalprogplan.2025.102647","url":null,"abstract":"<div><h3>Background</h3><div>Sport-for-development (SFD) is an innovative approach utilizing sport to foster positive physical, mental, and social outcomes among children and youth, particularly those from underserved backgrounds. Living labs, which emphasize participant-centered research conducted in natural, real-world environments, present unique challenges for outcome measurement, including reduced control over conditions, variability in participant engagement, and logistical issues that complicate standardized data collection. Further, there are few outcome measures that are developed for SFD measurement in living lab settings. For these reasons, outcome measurement in a living lab setting remains challenging.</div></div><div><h3>Objective</h3><div>Our objective was to evaluate a set of outcome measures that have been administered in a living lab setting to better understand their performance, reliability, and areas for improvement.</div></div><div><h3>Methods</h3><div>SFD programming was delivered in a living lab setting at a large facility located in an urban center in Toronto, Canada. We evaluated 11, self-reported, Likert-style outcome measures against 8 key metrics used in Classical Test Theory to understand (for example) floor-and-ceiling effects, inter-item correlations, internal consistency, and test-retest reliability. Data were collected from 2019 to 2024 across multiple cohorts aged 6–29 years, involving diverse SFD programs.</div></div><div><h3>Results</h3><div>Our analysis of 2656 questionnaire completions demonstrated strengths in data collection, including complete response rates with minimal missing data (91 % of outcome measures met missingness thresholds), yet also highlighted issues primarily related to single-item-endorsement and inter-item correlations (with 38 % and 19 % of outcome measures meeting these thresholds, respectively). These insights prompted iterative improvements to the evaluation tools, such as modifying Likert scale response formats to include more response categories (and thereby reducing the impact of binning of responses).</div></div><div><h3>Conclusions</h3><div>Evaluating our outcome measures provided insight into how they can be improved for administration in a living-lab setting. The results emphasize the need for context-appropriate tools to effectively capture nuanced SFD program impacts and underscore the importance of ongoing validation to improve both research quality and practical implementation in living lab environments.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":"112 ","pages":"Article 102647"},"PeriodicalIF":1.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144571576","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Implementation and evaluation of service learning at higher education institutions","authors":"Marie-Therese Schultes ,&nbsp;Daniel Graf ,&nbsp;Julia Holzer ,&nbsp;Barbara Schober ,&nbsp;Christiane Spiel","doi":"10.1016/j.evalprogplan.2025.102622","DOIUrl":"10.1016/j.evalprogplan.2025.102622","url":null,"abstract":"<div><div>Service learning courses enable students and lecturers at higher education institutions to take on social responsibility by collaborating with community partners. In this literature review that is complemented by empirical examples, we present advantages and challenges of this teaching approach as well as guidelines for developing concepts for a university-wide implementation and evaluation of service learning. Our article is informed by higher education, implementation and evaluation research and findings from the Third Mission project of the University of Vienna, Austria. Within the scope of the project, we conducted a multi-stakeholder workshop and an interview study with representatives from seven universities. We illustrate the service learning approach with two course examples from the University of Vienna’s psychology master’s programme. Then, we discuss how service learning can be systematically implemented at the university level using frameworks from implementation science and findings from an interview study. Results indicate that successful implementation of service learning at the university level is supported by long-term collaboration between the university and community partners and an institutional culture that visibly values students’ and lecturers’ additional efforts. Finally, we seek to advance the evaluation of higher education by presenting a comprehensive participatory approach to service learning assessment that actively involves diverse stakeholders, including students, faculty, and community partners and that can be applied across various disciplines. By integrating multiple perspectives, we offer a framework for capturing the multidimensional impacts of service learning on educational outcomes and community engagement.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":"112 ","pages":"Article 102622"},"PeriodicalIF":1.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144321771","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A multimodal analysis of resource allocation across U.S. cancer registries","authors":"Maggie Cole-Beebe ,&nbsp;Florence K.L. Tangka ,&nbsp;Jenny Beizer ,&nbsp;Amarilys Bernacet ,&nbsp;Stephen Brown ,&nbsp;Paran Pordell ,&nbsp;Reda Wilson ,&nbsp;Sandy Jones ,&nbsp;Sujha Subramanian","doi":"10.1016/j.evalprogplan.2025.102639","DOIUrl":"10.1016/j.evalprogplan.2025.102639","url":null,"abstract":"<div><div>This study assessed resource allocation among registry activities, which may provide insight for efficient collection of high-quality cancer incidence data. We used a multimodal approach and purposively sampled 21 participating population-based cancer registries in the United States to ensure variation across several registries. The registries reported prospective staffing data and retrospective costing data, completing data collection from October 2021 to September 2022, reporting retrospective costing data for July 1, 2020, through June 30, 2021. From lessons learned from prior studies, we engaged participating registries early and throughout the study, ensuring the collection of meaningful, accurate quantitative data, as well as insights not captured quantitatively. Case volume is a major driver of registry costs. (On average, high-volume registries outspend low-volume registries by nearly 3x, annually). Upon examination of registry activities by case volume, we found that the two most resource-intensive registry activities are data acquisition and data processing, which may be addressed by innovations, such as electronic reporting and automation. Innovative data transfer and processing approaches could increase timeliness of data collection and reduce the labor resources required to process manually collected data. Registries adopting these innovations might achieve cost savings, which could make resources available to support other registry activities.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":"112 ","pages":"Article 102639"},"PeriodicalIF":1.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144502703","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Protocol for the pragmatic evaluation of the Department of the Air Force’s Personal Financial Readiness Program","authors":"Catherine Walker O’Neal ,&nbsp;Mallory Lucier-Greer ,&nbsp;Evin W. Richardson ,&nbsp;Haley E. Miranda ,&nbsp;Allan Tate ,&nbsp;Brian K. McKay","doi":"10.1016/j.evalprogplan.2025.102645","DOIUrl":"10.1016/j.evalprogplan.2025.102645","url":null,"abstract":"<div><div>Federal policy mandates that United States military members receive numerous financial trainings across their career to improve their financial literacy and equip them to make informed financial decisions that, ultimately, promote positive individual and family well-being. Connected to this mandate, systematic evaluation efforts of the Department of the Air Force’s Personal Financial Readiness program were initiated to identify program outcomes and downstream results while simultaneously informing program improvements and modifications. When situated in a discussion of evaluation theory and design practices, documenting these outcome and process evaluation protocols can provide valuable insight for scientist-practitioners designing multi-site program evaluations in the context of real-world constraints, as is often the case for large institutions, including the military. The evaluation design includes a prospective cohort design for the <em>outcome</em> evaluation with surveys completed before and after training and 3- and 6-months following training. The surveys provide data on short-, mid-, and long-term outcomes. Data collected for the <em>process</em> evaluation, include targeted surveys and interviews with individuals conducting and participating in the training. Together, this protocol illustrates a theory-informed evaluation design of a complex, multi-site set of trainings utilizing diverse data collection methods to conduct a rigorous outcome and process evaluation.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":"112 ","pages":"Article 102645"},"PeriodicalIF":1.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144535974","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Exploring the critical risk factors in COVID-19 vaccination clinics aimed uncertainty: Insight from Pakistan","authors":"Hafiz Waqar Abbas ,&nbsp;Neelum Khalid ,&nbsp;Razia Anjum ,&nbsp;Rashed Nawaz ,&nbsp;Zhongliang Zhou ,&nbsp;Shaoqing Gong","doi":"10.1016/j.evalprogplan.2025.102651","DOIUrl":"10.1016/j.evalprogplan.2025.102651","url":null,"abstract":"<div><h3>Background</h3><div>One of the key lessons emerging from the COVID-19 pandemic is the necessity of assessing and strengthening the operational capacity of COVID-19 vaccination clinics (CVCs) to ensure effective response and service delivery during current and future pandemics. Countries with more robust and resilient CVC infrastructure were better equipped to manage mass vaccination efforts compared to those with constrained capacities. This study focuses on Pakistan, a lower-middle-income country, where mass vaccination campaigns across five regions encountered substantial operational challenges due to limited CVC capacity. Therefore, the main objective of this study is to develop a decision support framework and identify CVCs critical capacity factor criteria for regional risk ranking. Secondly, to analyse the negative impact of one most critical factor's weight change on other factors in each capacity domain and the ranking of five different regions under the uncertainty of the COVID-19 pandemic.</div></div><div><h3>Methods</h3><div>CVC's capacity domains and associated risk factors are identified from the literature, and a decision support framework is constructed based on the consensus of expert opinion through the Delphi technique. Further, secondary data is extracted from Pakistan’s national COVID-19 vaccination database. Overall, CVC's capacity risk evaluation process is comprised of the entropy weight method, grey rational analysis (GRA), and the new weight change method. The Shannon entropy weight method was used to determine the criteria weight. The GRA was based on the grey system theory. It was adopted for regional risk ranking and the sequential process. The new weight change method was selected for the sensitivity analysis.</div></div><div><h3>Results</h3><div>The study identified nine critical factor criteria for evaluating CVCs capacity: Operational status, Monitoring of CVCs, Standee/charts providing basic COVID-19 information, Availability of smart devices for data entry and reporting, Minimum number (0–50) of vaccinations per shift, Staff trained in cold chain management, Vaccine administration checklist, Staff trained to identify Adverse Events Following Immunization (AEFI), and Display of Healthcare Waste Management (HCWM) guidelines. The regional risk ranking revealed that Sindh and Khyber Pakhtunkhwa (KPK) were categorized as high-risk regions, while Baluchistan was assessed as moderate risk. In contrast, Punjab and other composite regional areas exhibited low risk, indicating relatively stronger preparedness and operational performance.</div></div><div><h3>Conclusions</h3><div>This research provides a comprehensive CVC capacity risk assessment. Findings from this research may provide key information that can be helpful in countries where health systems are facing severe CVC capacity constraints.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":"112 ","pages":"Article 102651"},"PeriodicalIF":1.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144633330","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A quasi-experimental approach to evaluating magnet schools","authors":"Jia Wang ,&nbsp;Rolf Straubhaar ,&nbsp;Seth Leon ,&nbsp;David Kikoler ,&nbsp;Elaine Rosales","doi":"10.1016/j.evalprogplan.2025.102666","DOIUrl":"10.1016/j.evalprogplan.2025.102666","url":null,"abstract":"<div><div>The purpose of this article is to document an innovative and effective quasi-experimental, mixed methods approach developed for evaluating the effectiveness of magnet schools throughout the United States. Magnet schools continue to be one of the most common school choice mechanisms in the United States, and as a research team the authors have been evaluating the effectiveness of magnet schools in improving student achievement since 2010. In this paper, the authors share the evaluation research design that they have used through this work, with the intent of sharing best practices in effective magnet school evaluation with other scholars and practitioners engaged in work in similar school settings. Specifically, the authors here explain the benefits of various elements of their evaluation design: their selection of comparison schools, of comparison students within those schools, and their means of measuring magnet program implementation to evaluate the degree to which implementation affected student outcomes.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":"112 ","pages":"Article 102666"},"PeriodicalIF":1.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144711104","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Relationship between geographical factors and digital skills development among school administrators: A case study of Pathum Thani Province, Thailand","authors":"Pimolpun Phetsombat ,&nbsp;Nawhath Thanwiset Thanvisitthpon","doi":"10.1016/j.evalprogplan.2025.102663","DOIUrl":"10.1016/j.evalprogplan.2025.102663","url":null,"abstract":"<div><div>This study investigates how geographical factors influence the digital skills of educational administrators in Thailand’s Pathum Thani province. A digital skills assessment framework, comprising three components: competency, knowledge, and experience, was developed and validated using confirmatory factor analysis. Data from 346 educational staff and 219 school teachers across seven districts were analyzed. The findings revealed significant district-level variation in digital skills, influenced by factors such as student population, student density, and budget support. The assessment framework demonstrated strong reliability and validity, with all model fit indices meeting accepted thresholds. This research contributes by providing a validated tool to assess digital proficiency in educational leadership and by identifying key geographical factors that shape digital skills development. It underscores the need for targeted digital training programs and equitable resource allocation to enhance digital proficiency among school administrators. These insights can inform national policies and guide the design of district-level digital literacy initiatives.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":"112 ","pages":"Article 102663"},"PeriodicalIF":1.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144702615","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A scoping review of program evaluations in non-profit alcohol and other drugs service settings: Considerations for service and research partnerships","authors":"Emily Deans ,&nbsp;George Economidis ,&nbsp;Peter Gates ,&nbsp;Andrea Zocco ,&nbsp;Robert Stirling ,&nbsp;Sally Nathan ,&nbsp;Suzie Hudson ,&nbsp;Sara Farnbach","doi":"10.1016/j.evalprogplan.2025.102614","DOIUrl":"10.1016/j.evalprogplan.2025.102614","url":null,"abstract":"<div><h3>Introduction</h3><div>The non-profit sector contributes significantly to preventing and reducing harm from alcohol and other drugs (AOD). However, it can experience challenges when demonstrating impact via feasible and rigorous evaluation.</div></div><div><h3>Methods</h3><div>We conducted a scoping review to identify characteristics of, and barriers/enablers to, evaluations completed in non-profit AOD services, and to ascertain the degree of collaboration between researchers and service providers during three evaluation phases (preparatory, conduct; and dissemination and implementation). We searched EMBASE and Medline from inception to February 2023 for evaluations completed in small to medium non-profit AOD services. Screening and extraction were completed by three reviewers independently, and 80 % were checked by an additional reviewer.</div></div><div><h3>Results</h3><div>Ultimately, 56 records were included in this study. Included evaluations were largely initiated by researchers, with limited input from service providers in setting the agenda for the evaluations (preparatory phase) or translating findings into practice (dissemination and implementation phase). Where collaborations between researchers and service providers were in place, they generated funding and strong engagement with participants, leading to better recruitment, more complete data collection, and enhanced evaluation outcomes.</div></div><div><h3>Conclusion</h3><div>The review raises important implications for researchers and service providers to prioritise meaningful collaboration and innovation to inform appropriate selection of evaluation topics and designs. Specifically, such evaluations should generate knowledge which can be translated into practice, demonstrating the impact of services and programs and an evidence base built in the non-profit sector.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":"112 ","pages":"Article 102614"},"PeriodicalIF":1.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144270663","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Development of a health promoting sports clubs intervention: An intervention mapping process analysis","authors":"Van Hoye ,&nbsp;Stacey Johnson ,&nbsp;Florence Rostan ,&nbsp;Fabienne Lemonnier ,&nbsp;Benjamin Tezier ,&nbsp;Susanna Geidne ,&nbsp;Alex Donaldson ,&nbsp;Anne Vuillemin","doi":"10.1016/j.evalprogplan.2025.102640","DOIUrl":"10.1016/j.evalprogplan.2025.102640","url":null,"abstract":"<div><div>Developing and implementing health promotion interventions in sports clubs is challenging. Implementing interventions in real world settings should be specific to a sports club’s context, particularly their voluntary nature. The present study maps and analyzes the development of the PROSCeSS intervention, designed to support sports clubs to implement health promotion. A case study design mobilising Intervention Mapping — needs assessment, matrix of change, theory of action, program design, plan for adoption and implementation and evaluation — describes the multi-phase intervention. Results encompass the main findings from i) a literature review and a concept mapping study as a needs assessment; ii) a Delphi study and health promoting sports club intervention framework to design the matrix of change; iii) a case study among sports clubs and sports federations to select the theory of action; iv) the intervention co-construction to create the program; v) results from a pilot study to validate the plan for adoption; and vi) implementation and presentation of the e-PROSCeSS questionnaire creation and hybrid type-3 protocol as an evaluation method. The discussion explores the tensions created between evidence driven strategies and the habits of sports clubs, the use of top-down versus bottom-up approaches and reflections on the recursive rather than linear process adopted by the authors.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":"112 ","pages":"Article 102640"},"PeriodicalIF":1.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144502702","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}