Evaluation Review最新文献

{"title":"The Logic of Generalization From Systematic Reviews and Meta-Analyses of Impact Evaluations.","authors":"Julia H Littell","doi":"10.1177/0193841X241227481","DOIUrl":"10.1177/0193841X241227481","url":null,"abstract":"<p><p>Systematic reviews and meta-analyses are viewed as potent tools for generalized causal inference. These reviews are routinely used to inform decision makers about expected effects of interventions. However, the logic of generalization from research reviews to diverse policy and practice contexts is not well developed. Building on sampling theory, concerns about epistemic uncertainty, and principles of generalized causal inference, this article presents a pragmatic approach to generalizability assessment for use with systematic reviews and meta-analyses. This approach is applied to two systematic reviews and meta-analyses of effects of \"evidence-based\" psychosocial interventions for youth and families. Evaluations included in systematic reviews are not necessarily representative of populations and treatments of interest. Generalizability of results is limited by high risks of bias, uncertain estimates, and insufficient descriptive data from impact evaluations. Systematic reviews and meta-analyses can be used to test generalizability claims, explore heterogeneity, and identify potential moderators of effects. These reviews can also produce pooled estimates that are not representative of any larger sets of studies, programs, or people. Further work is needed to improve the conduct and reporting of impact evaluations and systematic reviews, and to develop practical approaches to generalizability assessment and guide applications of interventions in diverse policy and practice contexts.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"427-460"},"PeriodicalIF":0.9,"publicationDate":"2024-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139543102","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Improving the Usefulness and Use of Meta-Analysis to Inform Policy and Practice.","authors":"Rebecca Maynard","doi":"10.1177/0193841X241229885","DOIUrl":"10.1177/0193841X241229885","url":null,"abstract":"<p><p>This chapter begins with an overview of recent developments that have encouraged and facilitated greater use of research syntheses, including Meta-Analysis, to guide public policy and practice in education, workforce development, and social services. It discusses the role of Meta-Analysis for improving knowledge of the effectiveness of programs, policies, and practices and the applicability and generalizability of that knowledge to conditions other than those represented by the study samples and settings. The chapter concludes with recommendations for improving the potential of Meta-Analysis to accelerate knowledge development through changing how we design, conduct, and report findings of individual studies to maximize their usefulness in Meta-Analysis as well as how we produce and report Meta-Analysis findings. The paper includes references to resources supporting the recommendations.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"515-543"},"PeriodicalIF":0.9,"publicationDate":"2024-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11003195/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139673299","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Transferability of Lessons From Program Evaluations: Iron Laws, Hiding Hands and the Evidence Ecosystem.","authors":"Tom Ling","doi":"10.1177/0193841X241228332","DOIUrl":"10.1177/0193841X241228332","url":null,"abstract":"<p><p>Assessing the transferability of lessons from social research or evaluation continues to raise challenges. Efforts to identify transferable lessons can be based on two different forms of argumentation. The first draws upon statistics and causal inferences. The second involves constructing a reasoned case based on weighing up different data collected along the causal chain from designing to delivery. Both approaches benefit from designing research based upon existing evidence and ensuring that the descriptions of the programme, context, and intended beneficiaries are sufficiently rich. Identifying transferable lessons should not be thought of as a one-off event but involves contributing to the iterative and learning of a scientific community. To understand the circumstances under which findings can be confidently transferred, we need to understand: (1) How far and why outcomes of interest have multiple, interacting and fluctuating causes. (2) The program design and implementation capacity. (3) Prior knowledge and causal landscapes (and how far these are included in the theory of change). (4) New and relevant knowledge; what can we learn in our 'disputatious community of truth seekers'.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"410-426"},"PeriodicalIF":0.9,"publicationDate":"2024-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139486569","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"How Mixed-Methods Research Can Improve the Policy Relevance of Impact Evaluations.","authors":"Burt S Barnow, Sanjay K Pandey, Qian Eric Luo","doi":"10.1177/0193841X241227480","DOIUrl":"10.1177/0193841X241227480","url":null,"abstract":"<p><p>This paper describes how mixed methods can improve the value and policy relevance of impact evaluations, paying particular attention to how mixed methods can be used to address external validity and generalization issues. We briefly review the literature on the rationales for using mixed methods; provide documentation of the extent to which mixed methods have been used in impact evaluations in recent years; describe how we developed a list of recent impact evaluations using mixed methods and the process used to conduct full-text reviews of these articles; summarize the findings from our analysis of the articles; discuss three exemplars of using mixed methods in impact evaluations; and discuss how mixed methods have been used for studying and improving external validity and potential improvements that could be made in this area. We find that mixed methods are rarely used in impact evaluations, and we believe that increased use of mixed methods would be useful because they can reinforce findings from the quantitative analysis (triangulation), and they can also help us understand the mechanism by which programs have their impacts and the reasons why programs fail.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"495-514"},"PeriodicalIF":0.9,"publicationDate":"2024-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139651808","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Validity Evidence for an Observational Fidelity Measure to Inform Scale-Up of Evidence-Based Interventions","authors":"Pamela R. Buckley, Katie Massey Combs, Karen M. Drewelow, Brittany L. Hubler, Marion Amanda Lain","doi":"10.1177/0193841x241248864","DOIUrl":"https://doi.org/10.1177/0193841x241248864","url":null,"abstract":"As evidence-based interventions are scaled, fidelity of implementation, and thus effectiveness, often wanes. Validated fidelity measures can improve researchers’ ability to attribute outcomes to the intervention and help practitioners feel more confident in implementing the intervention as intended. We aim to provide a model for the validation of fidelity observation protocols to guide future research studying evidence-based interventions scaled-up under real-world conditions. We describe a process to build evidence of validity for items within the Session Review Form, an observational tool measuring fidelity to interactive drug prevention programs such as the Botvin LifeSkills Training program. Following Kane’s (2006) assumptions framework requiring that validity evidence be built across four areas (scoring, generalizability, extrapolation, and decision), confirmatory factor analysis supported the hypothesized two-factor structure measuring quality of delivery (seven items assessing how well the material is implemented) and participant responsiveness (three items evaluating how well the intervention is received), and measurement invariance tests suggested the structure held across grade level and schools serving different student populations. These findings provide some evidence supporting the extrapolation assumption, though additional research is warranted since a more complete overall depiction of the validity argument is needed to evaluate fidelity measures.","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"11 1","pages":""},"PeriodicalIF":0.9,"publicationDate":"2024-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140833906","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Conditioning on the Pre-Test versus Gain Score Modelling: Revisiting the Controversy in a Multilevel Setting","authors":"Bruno Arpino, Silvia Bacci, Leonardo Grilli, Raffaele Guetto, Carla Rampichini","doi":"10.1177/0193841x241246833","DOIUrl":"https://doi.org/10.1177/0193841x241246833","url":null,"abstract":"We consider estimating the effect of a treatment on a given outcome measured on subjects tested both before and after treatment assignment in observational studies. A vast literature compares the competing approaches of modelling the post-test score conditionally on the pre-test score versus modelling the difference, namely, the gain score. Our contribution lies in analyzing the merits and drawbacks of two approaches in a multilevel setting. This is relevant in many fields, such as education, where students are nested within schools. The multilevel structure raises peculiar issues related to contextual effects and the distinction between individual-level and cluster-level treatments. We compare the two approaches through a simulation study. For individual-level treatments, our findings align with existing literature. However, for cluster-level treatments, the scenario is more complex, as the cluster mean of the pre-test score plays a key role. Its reliability crucially depends on the cluster size, leading to potentially unsatisfactory estimators with small clusters.","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"2012 1","pages":""},"PeriodicalIF":0.9,"publicationDate":"2024-04-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140612307","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Do Trade and Financial Cooperation Improve Environmentally Sustainable Development: A Distinction Between <i>de facto</i> and <i>de jure</i> Globalization.","authors":"Mehmet Akif Destek, İbrahim Halil Oğuz, Nuh Okumuş","doi":"10.1177/0193841X231181747","DOIUrl":"10.1177/0193841X231181747","url":null,"abstract":"<p><p>The adoption of growth strategies based on foreign trade, especially in the previous century when liberal policies began to dominate, is one of the main reasons for the increase in output and indirectly for environmental concerns. On the other hand, there are complex claims about the environmental effects of liberal policies and thus of globalization. This study intends to analyze the effects of global collaborations involving 11 transition economies that have completed the transition process on the environmentally sustainable development of these nations. In this direction, the effects of financial and commercial globalization indices on carbon emissions are investigated. The distinctions of globalization are used to distinguish the consequences of the two types of globalization. In doing so, the de facto and de jure indicator distinctions of globalization are used to differentiate the consequences of two types of globalization. In addition, the effects of real GDP, energy efficiency, and use of renewable energy on environmental pollution are dissected. For the main purpose of the study, the CS-ARDL estimation technique that allows cross-sectional dependency among observed countries is used to separate the short and long-run influences of explanatory variables. In addition, CCE-MG estimator is used for robustness check. According to the empirical findings, the economic growth and increasing energy intensity increases carbon emissions, but the increase in renewable energy consumption improves environmental quality. Furthermore, trade globalization does not have a significant impact on the environment in the context of globalization. On the other hand, the increase in de facto and de jure financial globalization indices results in an increase in carbon emissions, but de jure financial globalization causes more environmental damage. The harmful impact of de jure financial globalization on environmental quality suggests that the decreasing investment restrictions and international investment agreements of transition countries have been implemented in a manner that facilitates the relocation of investments from pollution-intensive industries to these countries.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"251-273"},"PeriodicalIF":0.9,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9957122","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Promoting Pro-Environmental Behavior in Policymaking: A Text-Mining Approach for Literature Review.","authors":"Md Abul Kalam Azad, Mehedi Hasan Ifti, Chowdhury Noushin Novera, Anh Ngoc Quang Huynh, Esra Sipahi Döngül","doi":"10.1177/0193841X231176869","DOIUrl":"10.1177/0193841X231176869","url":null,"abstract":"<p><p>The impact of pro-environmental behavior on policymaking has been an exciting area of research. While the relationship between pro-environmental behavior and policymaking has been explored in numerous studies, there needs to be more synthesis on this topic. This is the first text-mining study of pro-environmental effects in which policymaking is a significant factor. In response, this study, for the first time, takes a novel approach by using text mining in R programming to analyze 30 publications from the Scopus database on pro-environmental behavior in policymaking, highlighting major research themes and prospective research areas for future investigation. Results from text mining yielded 10 topic models, which are presented with a synopsis of the published research and a list of the primary authors, as well as a posterior probability via latent Dirichlet allocation (LDA). Additionally, the study conducts a trend analysis of the top 10 journals with the highest impact factor, considering the influence of each journal's mean citation. The study offers an overview of the impacts of pro-environmental behavior in policymaking, showing the most relevant and frequently discussed themes, introduces the scientific visualization of papers published in the Scopus database, and proposes future study directions. These findings can help researchers and environmental specialists better understand how pro-environmental behavior can be fostered more effectively through policymaking.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"370-398"},"PeriodicalIF":0.9,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9543356","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Evaluating the Stroke Risk of Patients using Machine Learning: A New Perspective from Sichuan and Chongqing.","authors":"Jin Zheng, Yao Xiong, Yimei Zheng, Haitao Zhang, Rui Wu","doi":"10.1177/0193841X231193468","DOIUrl":"10.1177/0193841X231193468","url":null,"abstract":"<p><p>Stroke is the leading cause of death and disability among people in China, and it leads to heavy burdens for patients, their families and society. An accurate prediction of the risk of stroke has important implications for early intervention and treatment. In light of recent advances in machine learning, the application of this technique in stroke prediction has achieved plentiful promising results. To detect the relationship between potential factors and the risk of stroke and examine which machine learning method significantly can enhance the prediction accuracy of stroke. We employed six machine learning methods including logistic regression, naive Bayes, decision tree, random forest, K-nearest neighbor and support vector machine, to model and predict the risk of stroke. Participants were 233 patients from Sichuan and Chongqing. Four indicators (accuracy, precision, recall and F1 metric) were examined to evaluate the predictive performance of the different models. The empirical results indicate that random forest yields the best accuracy, recall and F1 in predicting the risk of stroke, with an accuracy of .7548, precision of .7805, recall of .7619 and F1 of .7711. Additionally, the findings show that age, cerebral infarction, PM 8 (an anti-atrial fibrillation drug), and drinking are independent risk factors for stroke. Further studies should adopt a broader assortment of machine learning methods to analyze the risk of stroke, by which better accuracy can be expected. In particular, RF can successfully enhance the forecasting accuracy for stroke.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"346-369"},"PeriodicalIF":0.9,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10302426","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Using Ecometric Data to Explore Sources of Cross-Site Impact Variance in Multi-Site Trials.","authors":"David R Judkins, Gabriel Durham","doi":"10.1177/0193841X231175549","DOIUrl":"10.1177/0193841X231175549","url":null,"abstract":"<p><p>In 2003, Bloom, Hill, and Riccio (BHR) published an influential paper introducing novel methods for explaining the variation in local impacts observed in multi-site randomized control trials of socio-economic interventions in terms of site-level mediators. This paper seeks to improve upon this previous work by using student-level data to measure site-level mediators and confounders. Development of asymptotic behavior backed up with simulations and an empirical example. Students and training providers. Two simulations and an empirical application to data from an evaluation of the Health Professions Opportunity Grants (HPOG) Program. This empirical analysis involved roughly 6600 participants across 37 local sites. We examine bias and mean square error of estimates of mediation coefficients as well as the true coverage of nominal 95-percent confidence intervals on the mediation coefficients. Simulations suggest that the new methods generally improve the quality of inferences even when there is no confounding. Applying this methodology to the HPOG study shows that program-average FTE months of study by month six was a significant mediator of both career progress and long-term degree/credential receipt. Evaluators can robustify their BHR-style analyses by the use of the methods proposed here.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":" ","pages":"274-311"},"PeriodicalIF":0.9,"publicationDate":"2024-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9969498","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}