Development and preliminary validation of the evaluation use scale for evaluation systems (EUS-ES)

IF 2.3 Q1 EDUCATION & EDUCATIONAL RESEARCH

Educational Research and Evaluation Pub Date : 2023-08-18 DOI:10.1080/13803611.2023.2261904

Roberto Brazileio Paixão, Michael C. Rodriguez

{"title":"Development and preliminary validation of the evaluation use scale for evaluation systems (EUS-ES)","authors":"Roberto Brazileio Paixão, Michael C. Rodriguez","doi":"10.1080/13803611.2023.2261904","DOIUrl":null,"url":null,"abstract":"ABSTRACTThe usefulness of evaluation is critical. Evaluation use occurs when, from its results or process, decisions are made about the program, it changes people's mindsets, or persuasive or legitimation actions happen (instrumental, conceptual, and symbolic uses respectively). Few quantitative evaluation use studies have been conducted in recent years. Most existing evaluation use scales are outdated due to theoretical advances and were project or program specific, not designed for evaluation systems. This study was designed to develop the Evaluation Use Scale for Evaluation Systems (EUS-ES) and gather initial validity evidence. We collected and analyzed data from Brazilian professors and program coordinators who were consultants of the Brazilian Graduate Evaluation System. Validity evidence was collected based on instrument content and internal structure. CFA results supported a three-factor-correlated model with adequate score reliability estimates. The EUS-ES is valuable for diagnostic purposes to identify low/high evaluation uses in an evaluation system setting.KEYWORDS: Evaluation useinstrumental useconceptual usesymbolic usescale Disclosure statementNo potential conflict of interest was reported by the author(s).Additional informationFundingThis work was supported in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – Brasil (CAPES) under Grant 88887.373319/2019-00.Notes on contributorsRoberto Brazileio PaixãoRoberto Brazileiro Paixão is an associate professor of educational evaluation.Michael C. RodriguezMichael C. Rodriguez is a professor of quantitative methods in education and dean of the College of Education and Human Development.","PeriodicalId":47025,"journal":{"name":"Educational Research and Evaluation","volume":"441 1","pages":"0"},"PeriodicalIF":2.3000,"publicationDate":"2023-08-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Educational Research and Evaluation","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1080/13803611.2023.2261904","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

ABSTRACTThe usefulness of evaluation is critical. Evaluation use occurs when, from its results or process, decisions are made about the program, it changes people's mindsets, or persuasive or legitimation actions happen (instrumental, conceptual, and symbolic uses respectively). Few quantitative evaluation use studies have been conducted in recent years. Most existing evaluation use scales are outdated due to theoretical advances and were project or program specific, not designed for evaluation systems. This study was designed to develop the Evaluation Use Scale for Evaluation Systems (EUS-ES) and gather initial validity evidence. We collected and analyzed data from Brazilian professors and program coordinators who were consultants of the Brazilian Graduate Evaluation System. Validity evidence was collected based on instrument content and internal structure. CFA results supported a three-factor-correlated model with adequate score reliability estimates. The EUS-ES is valuable for diagnostic purposes to identify low/high evaluation uses in an evaluation system setting.KEYWORDS: Evaluation useinstrumental useconceptual usesymbolic usescale Disclosure statementNo potential conflict of interest was reported by the author(s).Additional informationFundingThis work was supported in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – Brasil (CAPES) under Grant 88887.373319/2019-00.Notes on contributorsRoberto Brazileio PaixãoRoberto Brazileiro Paixão is an associate professor of educational evaluation.Michael C. RodriguezMichael C. Rodriguez is a professor of quantitative methods in education and dean of the College of Education and Human Development.

查看原文本刊更多论文

评估系统的评估使用量表(EUS-ES)的开发和初步验证

摘要评价的有用性至关重要。评估使用发生在，从其结果或过程中，关于计划的决定，它改变了人们的心态，或者发生了说服或合法化的行为(分别是工具性的，概念性的和象征性的使用)。近年来进行的定量评价使用研究很少。由于理论的进步，大多数现有的评估使用尺度已经过时，并且是特定于项目或计划的，而不是为评估系统设计的。本研究旨在制定评估系统的评估使用量表(EUS-ES)并收集初步效度证据。我们收集并分析了巴西教授和项目协调员的数据，他们是巴西研究生评估系统的顾问。根据仪器内容和内部结构收集效度证据。CFA结果支持三因素相关模型，具有足够的评分信度估计。在评估系统设置中，EUS-ES对于识别低/高评价用途的诊断目的是有价值的。关键词:评价用途工具性用途概念性用途象征性用途披露声明作者未报告潜在的利益冲突。本研究得到了巴西高级医疗卫生组织(CAPES)资助项目88887.373319/2019-00的部分支持。作者简介roberto Brazileio paix，教育评估副教授。迈克尔·c·罗德里格斯，教育学定量方法教授，教育与人类发展学院院长。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Educational Research and Evaluation EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

3.00

自引率

0.00%

发文量

期刊介绍： International, comparative and multidisciplinary in scope, Educational Research and Evaluation (ERE) publishes original, peer-reviewed academic articles dealing with research on issues of worldwide relevance in educational practice. The aim of the journal is to increase understanding of learning in pre-primary, primary, high school, college, university and adult education, and to contribute to the improvement of educational processes and outcomes. The journal seeks to promote cross-national and international comparative educational research by publishing findings relevant to the scholarly community, as well as to practitioners and others interested in education. The scope of the journal is deliberately broad in terms of both topics covered and disciplinary perspective.