Journal of multidisciplinary evaluation最新文献

{"title":"Integration of the Program Evaluation Standards into an Evaluation Toolkit for a Transformative Model of Care for Mental Health Service Delivery","authors":"M. Snow, M. Berger, A. Jaouich, Mélanie Hood, A. Salmon","doi":"10.56645/jmde.v19i43.837","DOIUrl":"https://doi.org/10.56645/jmde.v19i43.837","url":null,"abstract":"Background: Stepped Care 2.0 (SC2.0) is a transformative model of mental health service delivery. This model was created by Stepped Care Solutions (SCS), a not-for-profit consultancy that collaborates with governments, public service organizations, and other institutions that wish to redesign their mental health and addictions systems of care. The SC2.0 model is based on 10 foundational principles and 9 core components that can be flexibly adapted to an organization’s or community’s needs. The model supports groups to reorganize and deliver mental health care in an evidence-informed, person-centric way. SCS partnered with evaluators from the Centre for Health Evaluation and Outcome Sciences (CHÉOS) to create a toolkit that provides evaluation guidance. The toolkit includes a theory of change, guidance on selecting evaluation questions and designs, and an evaluation matrix including suggested process and outcome metrics, all of which can be tailored to each unique implementation of the SC2.0 model. The objective of this resource is to support organizations and communities to conduct high-quality evaluations for the purpose of continuous improvement (a core component of the model of care) and to assess the model’s impact.\u0000Purpose: The purpose of this paper is to discuss the integration of the program evaluation standards (PES) into an evaluation toolkit for SC2.0.\u0000Setting: In this paper, we describe the toolkit development, focusing on how the PES were embedded in the process and tools. We explore how the integration of the PES into the toolkit supports evaluators to enhance the quality of their evaluation planning, execution, and meta-evaluation.\u0000Intervention: Not applicable\u0000Research Design: Not applicable\u0000Data Collection and Analysis: Not applicable\u0000Findings: In this paper, we describe the toolkit development, focusing on how the PES were embedded in the process and tools. We explore how the integration of the PES into the toolkit supports evaluators to enhance the quality of their evaluation planning, execution, and meta-evaluation.\u0000Keywords: program evaluation standards; evaluation; mental health","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-04-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45760172","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Editor's Note","authors":"Michael A. Harnar","doi":"10.56645/jmde.v18i42.857","DOIUrl":"https://doi.org/10.56645/jmde.v18i42.857","url":null,"abstract":"","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2022-12-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47203011","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Empowerment Evaluation of Programs Involving Youth","authors":"Sarah Heath, K. Moreau","doi":"10.56645/jmde.v18i42.711","DOIUrl":"https://doi.org/10.56645/jmde.v18i42.711","url":null,"abstract":"Background: Participatory and collaborative evaluation approaches, including Empowerment Evaluation (EE), are useful for evaluating programs involving youth. Empowerment evaluation involves stakeholders in the evaluation process through a set of structured steps. It is primarily concerned with empowering, illuminating, and building program beneficiaries’ self-determination. Given the emphasis that EE places on inclusivity of stakeholders, it appears to be a good fit for evaluating programs that involve youth. \u0000Purpose: To explore the extent to which evaluators use EE to evaluate programs involving youth as well as what factor(s) facilitate and hinder their use of EE in these programs. \u0000Setting: The study involved evaluators associated with the Collaborative, Participatory and Empowerment Evaluation and Youth-Focused Evaluation Targeted Interest Groups (TIGs) of the American Evaluation Association (AEA) who are involved in evaluating programs targeted at youth. \u0000Intervention: Not applicable. \u0000Research Design: We used a two-phase sequential mixed-methods research design. In Phase 1, we surveyed evaluators. In Phase 2, we interviewed a sample of evaluators from Phase 1.  \u0000Findings: In Phase 1, 41 (53.9%) respondents indicated not using EE to evaluate programs involving youth, 30 (39.5%) had used EE and 5 (6.6%) were unsure. Of those who used EE, they used it to teach youth program stakeholders about evaluation (n=8, 24.2%), produce more authentic results by engaging youth as experts of their lived experience (n=7, 21.2%) or produce more useful results for stakeholders to use (n=6, 18.2%), as well as other less popular reasons. In Phase 2, 12 interviewees raised five factors that facilitate or hinder the use of EE to evaluate programs involving youth including, evaluator perceptions, type of evaluation experience, evaluator knowledge and professional training, guidelines from organizations and funders, and stakeholders and time. Factors that some interviewees viewed as facilitators others viewed as hinderances. \u0000  \u0000Keywords: empowerment evaluation, program evaluation, youth-focused evaluation \u0000  \u0000 ","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2022-12-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41567352","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Competitive champions versus cooperative advocates","authors":"Alison Rogers, A. Gullickson, J. King, E. McKinley","doi":"10.56645/jmde.v18i42.721","DOIUrl":"https://doi.org/10.56645/jmde.v18i42.721","url":null,"abstract":"Abstract \u0000Background: Evaluation offers non-profit organizations an opportunity to improve their services, demonstrate achievements, and be accountable. The extant literature identifies individuals who can enhance the uptake of evaluation as evaluation champions. However, a paucity of detail is available regarding how to identify them and the support they require. \u0000Purpose: This research investigated the characteristics and motivations of evaluation champions and examined how they promoted and embedded evaluation in an organizational system. \u0000Setting: Australian human and social service non-profit organizations. \u0000Research design: Drawing upon the literature and social interdependence theory, the research took an interpretivist perspective to collaboratively generate knowledge about evaluation champions. The aim was to understand and develop a reconstruction of the characteristics of individuals. This article constitutes a component of a larger research project. \u0000Data Collection and Analysis: This research used purposive sampling to recruit champions working in Australian non-profit organizations, who were identified via descriptive criteria gleaned from a literature review. The research involved interviewing 17 champions, four of whom also participated in organizational case studies. Analysis of the semi-structured interviews and case studies generated information about the activities, strategies, motivations, and attributes of individuals who championed and advocated for evaluation. \u0000Findings: This article argues that evaluation advocates is a preferable descriptor when attempting to embed evaluation by cultivating mutually beneficial interactions and cooperative working relationships. This research defines evaluation advocates as individuals who motivate others and provide energy, interest, and enthusiasm by connecting evaluation with colleagues’ personal aspirations and the organizational goals to make judgements about effectiveness. This article includes a field guide to facilitate evaluation advocates’ identification, recruitment, support, and development.","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2022-12-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47585180","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"How did conservation agriculture go to scale?","authors":"Ricardo Ramirez, C. Neudoerffer, M. Salomons","doi":"10.56645/jmde.v18i42.703","DOIUrl":"https://doi.org/10.56645/jmde.v18i42.703","url":null,"abstract":"Background \u0000The Foodgrains Bank has an established record working in agriculture and food security with resource constrained, marginalized farmers in sub-Saharan Africa. The three outcome areas of the Scaling-Up Conservation Agriculture in East Africa (SUCA) Program were: the adoption of conservation agriculture systems, an enabling institutional environment, and the promotion of enabling policies. These program areas were expected to yield intermediate outcomes that, together, would lead to the ultimate outcome of improved food security and sustainable livelihoods for smallholder farming households in East Africa. This case study reports on the end-line evaluation of the five-year program. \u0000Purpose \u0000To illustrate the overlap between utilization-focused evaluation (UFE) and collaborative approaches to evaluation (CAE). The case study profiles an agricultural intervention, and explores how the evaluation design accommodated the systemic nature of the program. \u0000Setting \u0000Scaling-Up Conservation Agriculture in East Africa (SUCA) was a five-year program of the Canadian Foodgrains Bank implemented from 2015-2020 to expand the size and scope of Foodgrains Bank’s work in conservation agriculture in East Africa. The program supported local partners with a target of 50,000 male and female farmers practicing a minimum of 2 of 3 conservation agriculture principles, and to improve food security and sustainable livelihoods for 18,000 of these farmers’ households across three countries.  \u0000Research design \u0000The Foodgrains Bank was directly involved in the evaluation design through the definition of evaluation uses and key evaluation questions. Eleven implementing partners in East Africa were involved in primary data collection and some initial analysis. \u0000Data collection and analysis \u0000A mixed method approach was used combining quantitative, qualitative, and participatory / visual data collection tools. A robust, intersectional gender lens was applied to the data collection instruments in the form of gender disaggregated data collection and gender-focused questions across most data collection instruments.  \u0000Findings \u0000The collaborative process confirmed a sense of ownership by the primary evaluation users over the evaluation design. The evaluation design combined outcome and learning uses that took advantages of the implementing organizations’ commitment to learning. The findings demonstrated the value of the program and produced a framework illustrating the multi-disciplinary approach underlying its success.","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2022-12-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43697316","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Quality as praxis","authors":"Amy Jersild, Michael A. Harnar","doi":"10.56645/jmde.v18i42.697","DOIUrl":"https://doi.org/10.56645/jmde.v18i42.697","url":null,"abstract":"Summative meta-evaluation is known to be more commonly practiced than formative meta-evaluation. While evaluation theorists speak to the importance of formative meta-evaluation, examples of how to do this are rarely specified in the evaluation literature. This paper aims to (1) further explore formative meta-evaluation as a means for quality assurance, with implications for both developing the capacity of evaluators and for advancing evaluation as a field of practice; and (2) to present a model with the intent to move toward a more deliberate formative quality evaluation practice. Discussion focuses on the relationship between evaluator and commissioner and how the development and use of a deliberate approach to formative meta-evaluation, through examination of the proposed model, can lead to a more egalitarian and inclusive approach to defining and promoting evaluation quality. Lastly, formative meta-evaluation is discussed as an important tool for evaluators in exercising professional judgment and for taking an active role in advancing the evaluation field.","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2022-12-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42063247","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Intangible Outcomes","authors":"Kurt Wilson","doi":"10.56645/jmde.v18i42.735","DOIUrl":"https://doi.org/10.56645/jmde.v18i42.735","url":null,"abstract":"Human life – and therefore the scope of human goals – includes dimensions that are both visible and countable (e.g., money, weight, attendance or tested proficiency) as well the invisible and intangible (e.g., hope, trust, faith, love, joy, peace). Furthermore, the visible and tangible aspects of life are intrinsically connected to and dependent on the invisible and intangible aspects - much as the visible branches and fruit of a tree are connected to and dependent on an underlying and hidden root structure. While the importance of intangibles can be understood intuitively, it can also be illustrated: 73% of all chartable giving in the U.S. goes to organizations that are explicitly religious, and 118,280 nonprofit organizations are so strongly identified with the intangibles of hope, trust, faith, love, joy, peace that they included one of these words in their name. While the intangible realities of human life are explicitly relevant to a large proportion of organizations we seek to serve, it is essentially ignored by current evaluation practice: only 10 articles within the American Journal of Evaluation, New Directions in Evaluation and Journal of Multidisciplinary Evaluation included even minimal reference to the most common intangibles. New evaluation theory and methodology to address this gap will be needed, and cross-disciplinary exploration with psychology, philosophy and sociology should guide this development. In the meantime, useful questions about intangibles can be drawn from the AEA guiding principles and addressing these can provide a useful starting point for evaluators seeking to consider intangibles within their evaluations.  ","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":"76 8","pages":""},"PeriodicalIF":0.0,"publicationDate":"2022-12-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41309202","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Conditions to Consider in the Use of Randomized Experimental Designs in Evaluation","authors":"G. Julnes, M. Mark, Stephanie L. Shipman","doi":"10.56645/jmde.v18i42.741","DOIUrl":"https://doi.org/10.56645/jmde.v18i42.741","url":null,"abstract":"Debates about the role of randomized experiments in evaluation have been heated at times, which likely has not facilitated and possibly has hindered thoughtful judgments about whether and when to use a randomized experimental design. The challenges of thoughtful deliberation may be especially great for funders and others who influence the choice of an evaluation design but are not immersed in methodological literatures. The current paper offers a non-technical summary of general factors to take into consideration when determining the appropriateness of a randomized design in a forthcoming evaluation or set of evaluations. Four general conditions are described that should be considered with respect to the specific context for the upcoming evaluation(s). These are, first, the expected value of the information that a well-implemented experiment can provide in the specific context; second, the legal and ethical issues that apply in the circumstances at hand; third, the practical constraints (or facilitating factors) that would apply to a randomized experiment in that context; and fourth, the likely value of the experimental findings in relation to and as part of a portfolio of evaluative studies in the specific context.","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2022-12-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47472507","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Equity “On the Sideline”","authors":"Emily F. Gates, Eric Williamson, Joseph Madres, Kayla Benitez Alvarez, J. Hall","doi":"10.56645/jmde.v18i42.715","DOIUrl":"https://doi.org/10.56645/jmde.v18i42.715","url":null,"abstract":"Background: Centering equity in evaluations is increasingly recognized as an important professional responsibility of evaluators. While some theoretical and practical guidance exists, the evaluation field has limited empirical research on equity within evaluation practice. \u0000Purpose: This paper explores whether and how evaluators address inequities and advance equity throughout evaluation phases drawing on select findings from a larger study. \u0000Setting: The study focuses on American Evaluation Association-affiliated evaluators in the New England region of the United States who work in a variety of areas (e.g., health, education).   \u0000Intervention: Not applicable \u0000Research Design: The study uses a complementarity, sequential mixed methods design comprised of a researcher-developed online questionnaire administered to a census and snowball sample of practicing evaluators (n=82) and individual, semi-structured interviews with a subset of this sample selected to maximize variation (n=21). Quantitative data were analyzed using descriptive statistics (i.e., means and standard deviations, frequencies). Qualitative data were analyzed using a collaborative process of deductive and inductive coding followed by thematic analysis. \u0000Findings: Eight overarching findings suggest that despite evaluators’ attempts to center equity, it remains largely “on the sideline.” This is due to evaluators’ need to work against some conventional professional and methodological norms, within contractual and contextual constraints, and with limited professional preparation. \u0000 ","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2022-12-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46104311","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"New research frontiers pertaining to the infant gut microbiota.","authors":"Marco Ventura, Douwe van Sinderen, Francesca Turroni","doi":"10.20517/mrr.2022.12","DOIUrl":"10.20517/mrr.2022.12","url":null,"abstract":"<p><p>The human gut microbiota is believed to be responsible for multiple health-impacting host effects. The influence of gut microorganisms on the human host begins immediately after birth, having long-lasting health effects, while the gut microbiota itself continues to develop throughout the host's entire life. The purported health-associated effects of the gut microbiota have fueled extensive and ongoing research efforts. Nonetheless, the precise mode of action of functionalities exerted by microbial colonizers of the infant intestine is still largely unknown. The current perspective intends to illustrate major future investigative directions concerning the human gut microbiota with a specific focus on infant-associated gut microbes.</p>","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":"1 1","pages":"24"},"PeriodicalIF":0.0,"publicationDate":"2022-09-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10688817/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"88642434","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}