American Journal of Evaluation最新文献

{"title":"Measuring Evaluator Competencies: Developing and Validating the Evaluator Competencies Assessment Tool","authors":"Minji Cho, Ann Marie Castleman, Haley Umans, Mike Osiemo Mwirigi","doi":"10.1177/10982140211056539","DOIUrl":"https://doi.org/10.1177/10982140211056539","url":null,"abstract":"Evaluation scholars have committed decades of work to the development of evaluator competencies. The 2018 American Evaluation Association (AEA) Evaluator Competencies may be useful for evaluators to identify their strengths and weaknesses to improve their practice; however, a few empirically validated self-assessment tools based on the competencies exist. Two studies were conducted to develop a validated tool. The first study (N = 170) developed the Evaluator Competencies Assessment Tool (ECAT), a self-assessment tool based on the AEA, 2018 Evaluator Competencies. This study provided evidence for structural validity via confirmatory factor analysis. The second study (N = 142) reconfirmed structural validity with a new sample and examined variables that are associated with evaluator competencies through correlation and t-test analyses. Having a mentor, years of evaluation experience, age, evaluation training, and education level were positively related to evaluator competencies. The ECAT can be used to foster self-reflection for practitioners to improve evaluation competence.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":"44 1","pages":"474 - 494"},"PeriodicalIF":1.7,"publicationDate":"2022-12-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47459902","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The Process of Applying Principles-Focused Evaluation to the Sexual Violence Prevention Field: Implications for Practice in Other Social Services Fields","authors":"E. Casey, J. Vanslyke, B. Beadnell, N. Tatiana Masters, Kirstin McFarland","doi":"10.1177/10982140211056935","DOIUrl":"https://doi.org/10.1177/10982140211056935","url":null,"abstract":"Principles focused evaluation (PFE) can complement existing formative and outcome evaluation plans by identifying Effectiveness Principles (EPs), an operationalization of values and standards that guide practitioners during program implementation. To date, however, few examples of PFE are available in the literature. This description of the application of PFE to the Washington State Rape Prevention and Education (RPE) sexual violence prevention program provides an example of how this flexible approach can augment an existing evaluation plan to distill shared evaluation components across different organizations implementing diverse prevention programming. Specifically, we describe the process used by a team of practitioners, funders, evaluation consultants and state-level sexual violence prevention technical assistance providers to identify EPs, operationalize indicators for each EP, and develop and test an EP measurement approach. In this process, the seven very different RPE-funded organizations, each serving a unique community, were able to identify and endorse shared, core EPs. This description illustrates PFE's promise for augmenting a shared evaluation approach and identifying common guiding tenets across uniquely situated organizations in a larger community of practice.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":"44 1","pages":"374 - 393"},"PeriodicalIF":1.7,"publicationDate":"2022-12-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44700944","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Fidelity and Adaptation of Programs: Does Adaptation Thwart Effectiveness?","authors":"Kate L. Nolt, L. Leviton","doi":"10.1177/10982140221138604","DOIUrl":"https://doi.org/10.1177/10982140221138604","url":null,"abstract":"Evidence-based programs and grassroots programs are often adapted during implementation. Adaptations are often hidden, ignored, or punished. Although some adaptations stem from lack of organizational capacity, evaluators report other adaptations happen in good faith or are efforts to better fit the local context. Program implementers, facilitators who need to adapt during implementation, do not always report adaptations because they fear losing funding if the program is not implemented with fidelity. Program personnel including program evaluators need this information to improve effectiveness of programs, and to determine whether an adaptation is still consistent with the theory of change. Evaluators also need this information for generalizing results to varied settings and populations. Following the PRECEDE–PROCEED model, we recommend a hybrid approach to fidelity and adaptation. We argue in favor of advance planning to accommodate potential adaptations. Such planning also establishes evaluation criteria for determining whether adaptations are helpful, harmful, and appropriate to the context. We illustrate some types of adaptations that can occur, why they may be needed, and how to structure transparent reporting about adaptations to program developers and funding organizations.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":"44 1","pages":"322 - 334"},"PeriodicalIF":1.7,"publicationDate":"2022-12-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45018945","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Principles for Stakeholder Involvement in Evaluation in Latin America","authors":"Julia Espinosa-Fajardo, Pablo Rodríguez-Bilella, Esteban Tapella","doi":"10.1177/10982140221123010","DOIUrl":"https://doi.org/10.1177/10982140221123010","url":null,"abstract":"In the last three decades, the promotion of stakeholder involvement in evaluation has been gaining relevance in the Latin American and internationally, across varied agencies, institutions, and civic organizations. The 2030 Agenda and the Global Evaluation Agenda have also recognized the centrality of participation in evaluation. This article explores stakeholder involvement in evaluation based on collaborative work with stakeholders from 15 evaluative experiences. It shows what characterizes participatory evaluation in the region today and the principles of this practice.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":" ","pages":""},"PeriodicalIF":1.7,"publicationDate":"2022-12-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48929468","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Supporting Evaluation Practice Through Mindfulness","authors":"J. Pann, E. DiLuzio, A. Coghlan, Scott D. Hughes","doi":"10.1177/10982140221116094","DOIUrl":"https://doi.org/10.1177/10982140221116094","url":null,"abstract":"This article explores the utility of mindfulness in the field of evaluation. Mindfulness is a translation of the ancient Indian word, Sati, which means awareness, attention, and remembering. While definitions vary, a practical definition of mindfulness is present-moment awareness in an open and nonjudgmental manner. Mindfulness-based interventions have been employed by a wide variety of professions. Although it has received limited attention in the writings of evaluators, we argue that mindfulness can improve the practice of evaluation and support the development of the professional practice and interpersonal domains of American Evaluation Association (AEA) evaluator competencies. We review several mindfulness-based practices and how they can be used by evaluators in their work. Thus, we posit that far from being an esoteric concept, mindfulness practices can serve the pragmatic end of improving our discipline. We also discuss the limits of mindfulness and propose recommendations for future efforts.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":"44 1","pages":"293 - 307"},"PeriodicalIF":1.7,"publicationDate":"2022-12-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47726105","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Professionalizing Evaluation: A Time-Bound Comparison of the American Evaluation Association's Foundational Documents","authors":"S. Tucker, L. Stevahn, J. King","doi":"10.1177/10982140221136486","DOIUrl":"https://doi.org/10.1177/10982140221136486","url":null,"abstract":"This article compares the purposes and content of the four foundational documents of the American Evaluation Association (AEA): the Program Evaluation Standards, the AEA Public Statement on Cultural Competence in Evaluation, the AEA Evaluator Competencies, and the AEA Guiding Principles. This reflection on alignment is an early effort in the third step of professionalization: defining how to use and recognize evaluator competencies. The analysis intentionally focuses on content and reflects on the implications of the differences and similarities across documents. The comparison reveals important questions of interest at both the micro level (individual evaluator) and the macro level (evaluation). The article concludes with challenges, learnings, and proposed next steps of AEA's Professionalization and Competencies Working Group.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":"44 1","pages":"495 - 512"},"PeriodicalIF":1.7,"publicationDate":"2022-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44055415","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"From the Co-Editors: Building Evaluative Capacity to Examine Issues of Power, Address Sensitive Topics, and Generate Actionable Data","authors":"J. Hall, Laura R. Peck","doi":"10.1177/10982140221134238","DOIUrl":"https://doi.org/10.1177/10982140221134238","url":null,"abstract":"The fourth issue of volume 43 is our fi rst issue as Co-Editors-in-Chief of the American Journal of Evaluation. We have arrived at this point on our journey as Editors of AJE re fl ecting on our capacity as evaluators. While we are seasoned evaluators with decades of experience between us, we fi nd it is necessary to reexamine our role and capacity as evaluators and ask ourselves re fl ective questions such as What authority do we have as evaluators to address issues of power and privilege in the context of an evaluation? How do we determine if our evaluation approaches address vulnerable communities and sensitive topics respectfully? What analytic capacity do we have to produce valid and actionable evidence? And, what is within our capacity, as evaluators, to generate positive change for individuals, communities, and society? The articles we have assembled for this issues provide informed thinking on these and related topics based on the evaluation literature and other fi elds of study. Together, the discourse provided in the seven articles and three method notes in this issue will undoubtedly open up possibilities to re fl ect on and enhance your evaluative capacity as it has ours. The lead article in this issue, Critical Evaluation Capital (CEC): A New Tool for Applying Critical Race Theory to the Evaluand by Alice E. Ginsberg, centers issues of power in evaluation practice by presenting a tool to support critical evaluation approaches that challenge the notion of objectivity, consider evaluation a value-laden enterprise, and position the role of the evaluator as an agent for change. Informed by the lens of critical race theory and community cultural wealth, Ginsberg ’ s tool enhances the capacity of evaluators to pay attention to different types of power within the context of an evaluand. Speci fi cally, the CEC tool converts issues of power into several overlapping categories of “ capital. ” Each category is de fi ned and provides thought-provoking questions useful to explore our authority as evaluators and the role of power and privilege in an evaluation context. To conclude the article, Ginsberg retroactively applies the CEC tool to an evaluation. By","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":"43 1","pages":"464 - 467"},"PeriodicalIF":1.7,"publicationDate":"2022-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48654353","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Exploring NSF-Funded Evaluators’ and Principal Investigators’ Definitions and Measurement of Diversity, Equity, and Inclusion","authors":"A. Boyce, Tiffany L. S. Tovey, Onyinyechukwu Onwuka, J. R. Moller, Tyler Clark, Aundrea Smith","doi":"10.1177/10982140221108662","DOIUrl":"https://doi.org/10.1177/10982140221108662","url":null,"abstract":"More evaluators have anchored their work in equity-focused, culturally responsive, and social justice ideals. Although we have a sense of approaches that guide evaluators as to how they should attend to culture, diversity, equity, and inclusion (DEI), we have not yet established an empirical understanding of how evaluators measure DEI. In this article, we report an examination of how evaluators and principal investigators (PIs) funded by the National Science Foundation's Advanced Technological Education (ATE) program define and measure DEI within their projects. Evaluators gathered the most evidence related to diversity and less evidence related to equity and inclusion. On average, PIs’ projects engaged in activities designed to increase DEI, with the highest focus on diversity. We believe there continues to be room for improvement and implore the movement of engagement with these important topics from the margins to the center of our field's education, theory, and practice.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":"44 1","pages":"50 - 73"},"PeriodicalIF":1.7,"publicationDate":"2022-11-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45738847","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Comparison of Fidelity Implementation Frameworks Used in the Field of Early Intervention","authors":"Colombe Lemire, M. Rousseau, C. Dionne","doi":"10.1177/10982140211008978","DOIUrl":"https://doi.org/10.1177/10982140211008978","url":null,"abstract":"Implementation fidelity is the degree of compliance with which the core elements of program or intervention practices are used as intended. The scientific literature reveals gaps in defining and assessing implementation fidelity in early intervention: lack of common definitions and conceptual framework as well as their lack of application. Through a critical review of the scientific literature, this article aims to identify information that can be used to develop a common language and guidelines for assessing implementation fidelity. An analysis of 46 theoretical and empirical papers about early intervention implementation, published between 1998 and 2018, identified four conceptual frameworks, in addition to that of Dane and Schneider. Following analysis of the conceptual frameworks, a four-component conceptualization of implementation fidelity (adherence, dosage, quality and participant responsiveness) is proposed.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":"44 1","pages":"236 - 252"},"PeriodicalIF":1.7,"publicationDate":"2022-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47289679","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Just Give Me an Example! Exploring Strategies for Building Public Understanding of Evaluation","authors":"Sarah Mason","doi":"10.1177/10982140211061018","DOIUrl":"https://doi.org/10.1177/10982140211061018","url":null,"abstract":"Evaluators often lament that the general public does not understand what we do. Yet, there is limited empirical research on what the general public does know—and think—about program evaluation. This article seeks to expand our understanding in this domain by capturing views about evaluation from a demographically representative sample of the U.S population. This article also explores different strategies for describing program evaluation to the general public. Using an experimental design, it builds on previous research by Mason and Hunt, testing a set of hypotheses about how to enhance communication about evaluation. Findings suggest that public understanding of evaluation is indeed low, although two specific communication strategies—using well-known examples of social programs and including a why statement that describes the purpose of evaluation—can strengthen understanding among members of the public.","PeriodicalId":51449,"journal":{"name":"American Journal of Evaluation","volume":"44 1","pages":"549 - 567"},"PeriodicalIF":1.7,"publicationDate":"2022-11-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46576725","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}