Journal of multidisciplinary evaluation最新文献

{"title":"Mixed Methods Research in Designing an Instrument for Consumer-Oriented Evaluation","authors":"D. Gallant, N. Luthy","doi":"10.56645/jmde.v16i34.583","DOIUrl":"https://doi.org/10.56645/jmde.v16i34.583","url":null,"abstract":"\u0000 \u0000 \u0000 \u0000Background: The educational product market has been gradually shifting from primarily print to primarily digital content. Educators must make quick decisions when selecting materials that will assist students in their learning. \u0000Purpose: Purposes of this study were to describe the application of a two-stage sequential mixed-method, mixed-model design in designing an instrument for consumer-oriented evaluation and to describe implications of using mixed methods research in developing a rubric to evaluate prekindergarten through Grade 12 digital content. \u0000Setting: The Ohio State University, Columbus, OH. \u0000  \u0000 \u0000 \u0000Intervention: N/A. \u0000Research design: A two-stage sequential mixed-method, mixed-model design. \u0000Data collection & analysis: In Stage 1, a modified electronic Delphi survey technique was implemented with US geographically dispersed subject matter experts. In Stage 2, cross-sectional focus group interviews were conducted with local teachers, administrators, and textbook publishers. \u0000Findings: Inclusion of multiple perspectives and viewpoints from teachers, administrators, textbook publishers, and experts on importance, clarity, and appropriateness of criteria to evaluate digital content resulted in a final version of the rubric that can be used by teachers and administrators to evaluate digital content that supports students’ learning in prekindergarten through Grade 12. \u0000 \u0000 \u0000 \u0000","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-02-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48625107","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Review of the DAC Evaluation Criteria: Upgrading Efficiency","authors":"Joaquin De la Concha","doi":"10.56645/jmde.v16i34.587","DOIUrl":"https://doi.org/10.56645/jmde.v16i34.587","url":null,"abstract":"Background: Efficiency has remained for long time the Cinderella of evaluation practice. According to several meta-evaluations only a small number of evaluations include robust and meaningful assessments in this field. \u0000Purpose: Following the review of the Development Assistance Committee’s (DAC) evaluation criteria this article reviews the main causes of efficiency remaining in a secondary role in evaluation practices and proposes: 1) upgrading it through broadening the current economic definition of efficiency, and 2) proposing a set of four evaluation dimensions revolving around the principle of sound financial management, social perceived value, result-based budgeting and interconnectedness with other criteria. \u0000Setting: Not applicable. \u0000Intervention: Not applicable. \u0000Research design: Literature review and empirical pilot testing of proposed evaluation methodology. \u0000Data collection & analysis: Qualitative analysis. \u0000Findings: This paper calls for a shift in focus from efficiency to a broader enriched principle of sound financial management including a strong partner-based focus and highlighting management as a pre- condition “sine qua non” for an intervention to be efficient. Efficiency assessments would then be twofold: managerial (procedures, policies and practices that lay out the requirements for efficiency to take place) and substantial (context, circumstances and reasoning for the existing budget balance between results and target groups). The key aspects of the proposal involve four proposed dimensions to be analysed under this renewed criteria that imply audits and evaluations come much closer and look at each other. \u0000Keywords: DAC evaluation criteria; efficiency; economy; equity; sound financial management; performance audit.","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-01-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46312627","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Equity Implications in Evaluating Development Aid: The Italian Case","authors":"M. Forestieri","doi":"10.56645/jmde.v16i34.537","DOIUrl":"https://doi.org/10.56645/jmde.v16i34.537","url":null,"abstract":"Background: In the field of development aid, social equity is an emerging issue that concerns the evaluation community in its theoretical and practical dimensions. \u0000Purpose: A widely held belief is that evaluators do not apply theory. In this paper, we intend to verify this statement about equity in the field of cooperation projects. \u0000Setting: Not applicable. \u0000Intervention: Not applicable. \u0000Research Design: We considered equity-focused approaches and found three common factors: stakeholder participation, attention to context, and focus on marginalized groups. These elements operate as screening criteria in identifying equity issues in a case study. \u0000Data Collection and Analysis: The paper examines a practical experience of Italian cooperation. This involved a review of evaluations reports completed between 2013 and 2014. The reports are analyzed according to the three screening criteria. \u0000Findings: The use of the three criteria has proved its worth in grasping the issues of equity neglected and often not recognized in reports. Once again a gap emerges between theory and practice. The availability of theoretical approaches is not sufficient. The paper, therefore, proposes a reflection on the responsibility of evaluation towards social justice. \u0000Keywords: social equity; development aid; theory and practice gap.","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-01-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46613118","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Learning How to Learn in Sustainability Transitions Projects: The Potential Contribution of Developmental Evaluation","authors":"Andrew S. Mitchell, M. Lemon","doi":"10.56645/jmde.v16i34.531","DOIUrl":"https://doi.org/10.56645/jmde.v16i34.531","url":null,"abstract":"Background: Community-based sustainability transitions projects are increasingly being considered for their potential as policy delivery vehicles for the UK government Climate Change Act commitments. At the same time, project funders seek reassurances that their investments are relevant in helping communities mitigate, and adapt to, the effects of climate change. Despite this increased pressure, recent research suggests that evaluations of such sustainability transitions projects have, on the one hand variable impacts, or impacts that are of an inadequate scale, duration, or type, or on the other, that project staff lack the capacity or resources to undertake monitoring and evaluation to the degree of rigour expected by policy makers and funders. \u0000Purpose: This article reports on an extended case study of a fully-funded five year community-based sustainability transitions project in Leicestershire, England. In particular, it reviews the deployment of developmental evaluation (DE) methods in an attempt to capture the project team’s learning about doing community-based sustainability work. \u0000Setting: A funded community-based sustainability transitions project in a south Leicestershire market town. \u0000Intervention: Developmental evaluation methods were used to capture project-based learning as a resource for project innovation and adaptation. \u0000Research Design: Phronetic case study. \u0000Data Collection and Analysis: Participant-observation, action research, focus and special issue group facilitation. \u0000Findings: Use of a developmental evaluation method identified key learning points for the project actors; these focused on how the project had adapted to the complexities of the operating environment through innovations in second-order learning or learning how to learn. The paper makes recommendations for the design and funding arrangements of community-based sustainability transitions initiatives and developmental evaluation is endorsed as a viable and promising adjunct to more traditional impact, economic, and process evaluation methodologies. \u0000Keywords: developmental evaluation; second-order learning; community-based sustainability; phronesis; action research; project design.","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-01-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42423230","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Method for Using Rubric Ratings on Fishbone Diagrams to Compare Case Studies","authors":"M. Davey, J. Morell","doi":"10.56645/jmde.v16i34.581","DOIUrl":"https://doi.org/10.56645/jmde.v16i34.581","url":null,"abstract":"Background: In multi-case study program evaluations, the large amount of qualitative data that are generated from interviews can be difficult to utilize. This is particularly so when inference must be made as to why some cases succeed and some fail. \u0000Purpose: This paper shows a method for comparing multiple evaluation sites by using a rubric to define ratings for relevant factors, and an Ishikawa fishbone diagram as a model to show relationships among those factors. We show how this technique identified reasons for differences in outcomes among the sites. \u0000Setting: The evaluation setting was a large-scale safety innovation in the U.S. railroad industry. Four cases were considered—two passenger railroads and two freight railroads. \u0000Intervention: The Confidential Close Call Reporting System (C3RS) program allowed railroad workers to confidentially submit “close calls” which were reviewed by a team made up of labor, management, and the Federal Railroad administration to determine ways to improve safety. \u0000Research design: Multiple comparative case study, Ishikawa root and contributing cause modeling, evaluative rubric scoring, and data visualization techniques. \u0000Data collection & analysis: Interview data were collected from four pilot railroad sites, each of which participated in a five-year test of C3RS. Testing periods overlapped, with the entire evaluation lasting about 12 years. \u0000Findings: The method of using Ishikawa fishbone diagrams with ratings from an evaluative rubric was an effective method to summarize, analyze, and present large quantities of qualitative data. The approach succeeded in explaining degrees of success and failure across the sites. The sponsor and industry stakeholders were able to understand the analysis and the findings, and to develop deep insight into how to promote successful implementation. \u0000Keywords: Multiple comparative case studies; qualitative methodology; qualitative coding; data visualization; fishbone diagrams; Ishikawa diagrams; evaluative rubrics; close calls; near miss; data visualization.","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":"1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2020-01-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41564190","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Big Shoes to Fill: An Evaluation Journey in the Footsteps of Daniel L. Stufflebeam","authors":"Sherrie-Ann Camilli","doi":"10.56645/jmde.v15i33.609","DOIUrl":"https://doi.org/10.56645/jmde.v15i33.609","url":null,"abstract":"\u0000 \u0000 \u0000 \u0000Background: Evaluation has evolved remarkably since the early 1960s, largely due to the innovative contributions of Daniel Stufflebeam and his colleagues. As a pioneer of evaluation methods, some of the notable achievements arising from Stufflebeam’s work include the context-input-process-product (CIPP) model, evaluation standards, and evaluation checklists.   \u0000  \u0000Purpose: The purpose of this paper is to explore Daniel Stufflebeam’s journey beginning with the early days of evaluation through to his retirement and unfortunate passing at 80 years old in 2017. Key features of the CIPP model are considered within a context of other popular models for comparison with the goal of finding relevance for use of CIPP evaluation in education settings. \u0000  \u0000 \u0000 \u0000Setting: Not applicable. \u0000  \u0000Intervention: Not applicable. \u0000  \u0000Research Design: Literature review. \u0000  \u0000Data Collection and Analysis: Not applicable. \u0000  \u0000Findings: Stufflebeam’s CIPP model and evaluation standards remain prominent in evaluation practices and his legacy will lay the foundation for future evaluators through continued professional development. \u0000  \u0000 \u0000 \u0000 \u0000","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":"1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-12-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42007178","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"What, How, and Why? A Comparative Analysis of 12 Goal-Free Evaluations","authors":"Brandon W. Youker","doi":"10.56645/jmde.v15i33.444","DOIUrl":"https://doi.org/10.56645/jmde.v15i33.444","url":null,"abstract":"\u0000 \u0000 \u0000 \u0000Background: Goal-free evaluation (GFE) is any evaluation in which the evaluator conducts the evaluation without reference to predetermined goals or objectives.   \u0000  \u0000Purpose: The purpose of this study is to examine GFE in actual practice focusing on what GFE is, how it is conducted, and why the evaluators use it. \u0000  \u0000Setting: Not applicable. \u0000  \u0000Intervention: Not applicable. \u0000  \u0000Research Design:  Document analysis. \u0000 \u0000 \u0000Data Collection and Analysis: The researcher collected data from a non-random sample of 12 cases of GFE found in published and non-published papers, reports, and guidebooks. The researcher analyzed the documents using quantitative and qualitative content analysis. \u0000  \u0000Findings: The findings suggest that goal-free evaluators consider GFE an outcome evaluation that supplements GBE. These goal-free evaluators typically used an ex post facto evaluation design, non-random sampling of stakeholders, and semi-structured interviewing to collect data. The evaluators described using GFE to improve the evaluand, to find side-effects, and to evaluate highly complex evaluands. \u0000  \u0000 \u0000 \u0000 \u0000","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48377414","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Retrospective Pretest and Counterfactual Self-Report: Different or Same?","authors":"Tony C. M. Lam, Edgar Valencia","doi":"10.56645/jmde.v15i33.575","DOIUrl":"https://doi.org/10.56645/jmde.v15i33.575","url":null,"abstract":"\u0000 \u0000 \u0000 \u0000Purpose: To examine discriminant validity of treatment participants’ self-report of the state they would be in had they not received treatment (counterfactual); specifically, the distinction between self-report of counterfactual and self-report of preintervention state (retrospective pretest). \u0000  \u0000Setting: An education department of a large University in North America. \u0000  \u0000Intervention: Methods of self-reporting research self-efficacy with counterfactual items and with retrospective pretest items. \u0000  \u0000Research design: A randomized comparison group design with two treatments that were defined by the version of the survey used in each. In the survey for the counterfactual condition, items about research self-efficacy without the influence of their program of studies were included. The survey in the retrospective pretest condition contained items regarding research self-efficacy before participating in their program of study. The same items about research self-efficacy at the current time (posttest) were included in both treatment conditions. \u0000  \u0000 \u0000 \u0000Data collection & analysis: Participants were graduate students recruited via email who answered an online survey about research self-efficacy. These students were randomly assigned to one of the two aforementioned treatments. Responses were analyzed using a mixed 2 by 2 randomized factorial ANOVA design with self-report method (counterfactual or retrospective pretest) as the between-subjects factor and time (pre and post intervention) as the within-subjects factor. \u0000  \u0000Findings: Our findings show that counterfactual and retrospective pretest scores and treatment effects computed based on these two sets of scores are virtually identical, casting doubt on participants’ ability to differentiate between a state of no treatment and a state at treatment commencement after they have received treatment. \u0000 \u0000 \u0000 \u0000","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-11-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44026588","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Using Program Theory to Evaluate a Graduate College Student Development Program","authors":"Shannon Gus, Ashley Keener, C. Bullard, Sarah Gordon","doi":"10.56645/jmde.v15i33.559","DOIUrl":"https://doi.org/10.56645/jmde.v15i33.559","url":null,"abstract":"\u0000 \u0000 \u0000 \u0000Background: The “3 Minute Presentation” is a graduate student competition based off the more popular “3 Minute Thesis” competition. The program aims to help graduate students learn to inform others of their research in a quick and accessible manner. Programs to engage graduate students more deeply in their education require evaluation to determine if they are useful and effective at meeting their intended goals. Evaluation literature in graduate educational programs is currently limited, but increasingly needed for both the field and the students served. \u0000  \u0000Purpose: Development and testing of a program-theory evaluation to understand participation, recruitment, preparation, training, skills, and confidence of graduate students engaging in a “3 Minute Presentation” competition at a state university. \u0000  \u0000Setting: Institution of Higher Education \u0000  \u0000  \u0000 \u0000 \u0000Intervention: 3 Minute Presentation competition \u0000  \u0000Research Design:  Mixed-method program-theory evaluation \u0000  \u0000Data Collection and Analysis: Direct observations and closed-ended survey analyzed through qualitative coding, descriptive statistics, group comparisons, and correlation analysis.   \u0000  \u0000Findings: Overall, the program evaluation found, with a possible lack of diversity in participants, that the program components of recruitment, preparation, and skill development work as expected.  Additionally, engagement in preparation was associated with competition scores and the perceived helpfulness of preparation was related to students’ confidence in their presentation skills. This evaluation was deemed useful for program improvement and capacity building in the program’s continuation at the university.  \u0000 \u0000 \u0000 \u0000","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-10-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47183615","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Children’s Opinion of Retrospective Pre-Post ‘Then-Test’ Survey Validity","authors":"Leanne M. Kelly","doi":"10.56645/jmde.v15i33.535","DOIUrl":"https://doi.org/10.56645/jmde.v15i33.535","url":null,"abstract":"\u0000 \u0000 \u0000 \u0000Background: Over the past forty years there have been a number of studies conducted to compare traditional pre-post surveys (pretest-posttests; administered in two stages before and after an intervention) with retrospective pre-posts (thentests or pre-then-post-tests; administered after intervention only, with participants asked to reflect back to complete the ‘pre’ retrospectively). These previous studies have been with adult respondents and overwhelmingly quantitative.   \u0000  \u0000Purpose: This paper examines children’s perspectives regarding traditional and retrospective pre-post self-report subjective surveys. \u0000  \u0000Setting: A school-based program run by a community services organisation in southeastern Melbourne. \u0000  \u0000Intervention: Both pre-post survey types were administered to sixty children attending a pro-social skills group run by a community services organisation in southeast suburban Melbourne. \u0000  \u0000 \u0000 \u0000Research Design:  Twenty children participated in eight small focus groups after completing the surveys. Each focus group was guided by three semi-structured questions, ran for 10-15 minutes, and had 2-3 participants. This research included an observation component as the researcher was present at the final session when the post surveys were completed. The research also utilised the quantitative findings from the surveys to check alignment with findings from the extant literature. \u0000  \u0000Data Collection and Analysis: Focus groups & qualitative analysis \u0000  \u0000Findings: The traditional and retrospective surveys confirm that the commonly recorded phenomenon of response shift in adults also occurs with children. Children comment that they prefer the retrospective test, identifying concerns that support theories discussed in the extant literature such as experience limitation, impression management, implicit theory of change, and memory recall. \u0000  \u0000 \u0000 \u0000 \u0000","PeriodicalId":91909,"journal":{"name":"Journal of multidisciplinary evaluation","volume":" ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2019-07-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42404901","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}