Cherie M. Avent, Aileen Reid, J. R. Moller, Adeyemo Adetogun, Brianna Hooks Singletary, A. Boyce
{"title":"Navigating the Field While Black: A Critical Race Analysis of Peer and Elder Advice to and From Black Evaluators","authors":"Cherie M. Avent, Aileen Reid, J. R. Moller, Adeyemo Adetogun, Brianna Hooks Singletary, A. Boyce","doi":"10.1177/10982140231184902","DOIUrl":"https://doi.org/10.1177/10982140231184902","url":null,"abstract":"The field of evaluation has experienced greater professionalization in the areas of evaluator education and training. Growth in these areas included sensitivity to issues of diversity, with efforts to attract and retain evaluators of color. Currently, there is limited scholarship on navigating a world with more opportunity but still dealing with being Black. Motivated by our identities as Black evaluators, we explored the education, training and socialization concerns experienced by Black evaluators and the advice they would offer. Through content analysis of semi-structured interviews and the utilization of Critical Race Theory as a framework, we garnered advice to and from Black evaluators on how to (a) navigate the politics in evaluation; (b) engage in reflection and reflexivity; (c) network and collaborate; (d) mentor and bring others along the way; and (e) develop as a professional evaluator. We offer a synthesis of these findings in the discussion and articulate further implications.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2023-09-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44418390","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Translating Evaluation Policy Into Practice in Government Organizations.","authors":"Isabelle Bourgeois, Stéphanie Maltais","doi":"10.1177/10982140221079837","DOIUrl":"10.1177/10982140221079837","url":null,"abstract":"<p><p>Organizational evaluation policies describe how evaluation practices should be structured and implemented. As such, they provide key insights into organizational priorities and values regarding evaluation. However, the link between evaluation policies and how evaluation policies translate into concrete practices has seldom been explored until now. Our study examines the implementation of two Canadian federal government evaluation policies over a 10-year timespan, through the secondary analysis of reports produced on behalf of governmental evaluation functions. Our findings show that some policy elements have been fully implemented, but the way in which these have been implemented varies between organizations. Further, we observed that the level of control of various organizational members responsible for implementing policy elements, as well as time, can influence implementation of certain policy requirements. We conclude by proposing further directions for research to examine the policy-practice link.</p>","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/88/98/10.1177_10982140221079837.PMC10499548.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10287493","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
D. Follmer, Randall E. Groth, Jennifer A. Bergner, Starlin Weaver
{"title":"Theory-based Evaluation of Lesson Study Professional Development: Challenges, Opportunities, and Lessons Learned","authors":"D. Follmer, Randall E. Groth, Jennifer A. Bergner, Starlin Weaver","doi":"10.1177/10982140231184899","DOIUrl":"https://doi.org/10.1177/10982140231184899","url":null,"abstract":"“Lesson study” has garnered considerable attention from educational researchers and practitioners as a promising method for improving instruction. At its core, lesson study reflects a collaborative inquiry process, grounded in cycles of planning, enacting, and reflecting, that promotes teacher learning through active engagement in lesson refinement. In this article, we leverage lesson study as an exemplar case to foreground an examination and application of theory-based evaluation. We overview a novel evaluation framework contextualized to this multidimensional instructional program and draw out the contributions of theory-based evaluation to the study of program effects. To ground the description of this framework, we summarize representative findings from our ongoing evaluation of a lesson study-based instructional program and discuss processes for selecting, using, and managing evidence sources as well as determining causal strands supporting program outcomes. We conclude by discussing lessons learned and implications for theory-based evaluation of complex programs.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2023-08-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44960599","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Promoting Children's Rights and Protections in Program Evaluation","authors":"Duncan J. Mayer, V. Groza","doi":"10.1177/10982140221134621","DOIUrl":"https://doi.org/10.1177/10982140221134621","url":null,"abstract":"The Nuremberg Code established ethics for the involvement of humans in research, initially in the area of health and medical research. While aspects of the code have been extended to the social and behavioral sciences, program evaluation does not always implement those policies, procedures, and protocols for protecting research participants, particularly children. Drawing on a children's rights framework and highlighting participatory and trauma-informed approaches, this article explores the areas of consent, assent, cultural considerations, protocol/data collection procedure review, and participant safety. It proposes recommendations that fit a range of evaluation settings and resource level.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2023-07-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42594606","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Book Review: Evaluating and Valuing in Social Research by Thomas A. Schwandt and Emily F. Gates","authors":"Melvin E. Hall","doi":"10.1177/10982140231186518","DOIUrl":"https://doi.org/10.1177/10982140231186518","url":null,"abstract":"","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2023-07-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46364755","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Evaluation Paradoxes: Responding to Tensions Between Stability and Change in Social Investment Evaluation","authors":"Kettil Nordesjö","doi":"10.1177/10982140231185741","DOIUrl":"https://doi.org/10.1177/10982140231185741","url":null,"abstract":"The relationship between stability and change is a central paradox of administration that pervades all forms of organizing. Evaluation is not unfamiliar with paradoxical objectives and roles, which can result in tensions for evaluators and stakeholders. In this article, paradoxes between stability and change in the implementation of evaluation, and responses to them, are investigated through the case of social investment funds in Swedish local government. From interviews with staff, managers, and evaluators, findings show how responses to four main paradoxes give priority to top-down summative evaluation that produces instrumental knowledge on outcomes and costs for decision makers. The responses show that the concept of social investment fund evaluation is elastic to contain paradoxes and address different audiences. Also, paradoxes within the structure of the organization develop into paradoxes concerning the roles and goals of evaluation, raising the question of whether individual actors can deal with paradoxes.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2023-07-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46659794","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"From the Co-Editors: Policy, Politics, Principles, and Participation: Influences on Program Planning, Implementation, and Outcomes Achieved","authors":"J. Hall, Laura R. Peck","doi":"10.1177/10982140231184312","DOIUrl":"https://doi.org/10.1177/10982140231184312","url":null,"abstract":"Issue 3 of the American Journal of Evaluation ’ s Volume 44 has two parts. The fi rst collects a set of articles that consider Policy, Politics, Principles, and Participation: In fl uences on Program Planning, Implementation, and Outcomes Achieved . The second is a Special Section dedicated to the topic of evaluator competencies and the professionalization of our fi eld. A separate Special Section Editors ’ Note contextualizes that section, with a guest editor joining us for greater insights. This From the Co-Editors synthesizes the contributions of the fi rst half of the double issue. As always, it is our hope that AJE readers will fi nd value in the collection and use the articles to improve their practice. To begin, program implementation commonly involves adaptions to original program planning in response to changes in the program ’ s or","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2023-07-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41295554","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Special Section Editors’ Note: A Focus on the Evaluation Profession","authors":"Susan Tucker, Laura R. Peck, J. Hall","doi":"10.1177/10982140231184313","DOIUrl":"https://doi.org/10.1177/10982140231184313","url":null,"abstract":"Two of us (Hall and Peck) have been coeditors to the American Journal of Evaluation for over a year. During this time","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2023-07-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41757306","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Travis R. Moore, L. C. Valmadrid, Robyn Baragwanath, Nathaniel Haack, L. Bakken
{"title":"Gatekeeping's Influence on Equitable Evaluation Practice","authors":"Travis R. Moore, L. C. Valmadrid, Robyn Baragwanath, Nathaniel Haack, L. Bakken","doi":"10.1177/10982140231154503","DOIUrl":"https://doi.org/10.1177/10982140231154503","url":null,"abstract":"The ethical guidelines for the American Evaluation Association and the principles of community-based participatory evaluation both state the importance of equitable stakeholder involvement. Regardless of the evaluation approach, however, evaluators are often confronted with gatekeepers, or those who control the access to stakeholders, information, or resources. Gatekeepers limit both the participation of key community members and, therefore, the exchange of relevant information related to the evaluation—a process called gatekeeping. Little research attention has been placed on studying gatekeeping, resulting in a dearth of knowledge about the influence of gatekeeping on stakeholder-engaged evaluations and social-structural dynamics that potentially perpetuate gatekeeping practices. In this article, we propose a gatekeeping influence theory grounded in the findings from 14 interviews. With a constructed theory of gatekeeping, we document the emergent social-structural and relational dynamics involved in stakeholder-engaged evaluation, with a focus on evaluations that include community partners and members.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2023-06-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45195614","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Do Social Programs Help Some Beneficiaries More Than Others? Evaluating the Potential for Comparison Group Designs to Yield Low-Bias Estimates of Differential Impact","authors":"Andrew P. Jaciw","doi":"10.1177/10982140231160561","DOIUrl":"https://doi.org/10.1177/10982140231160561","url":null,"abstract":"In the current socio-political climate, there is an extra urgency to evaluate whether program impacts are distributed fairly across important student groups in education. Both experimental and quasi-experimental designs (QEDs) can contribute to answering this question. This work demonstrates that QEDs that compare outcomes across higher-level implementation units, such as schools, are especially well-suited to contributing evidence on differential program effects across student groups. Such designs, by differencing away site-level (macro) effects, on average produce estimates of the differential impact that are closer to experimental benchmark results than are estimates of average impact based on the same design. This work argues for the importance of routine evaluation of moderated impacts, describes the differencing procedure, and empirically tests the methodology with seven impact evaluations in education. The hope is to encourage broader use of this design type to more-efficiently develop the evidence base for differential program effects, particularly for underserved students.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":null,"pages":null},"PeriodicalIF":1.7,"publicationDate":"2023-06-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42773121","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}