Direct measurement of learning outcomes in higher education: A proposal of nine standardized scales for continuous improvement in engineering programs

IF 2 4区社会学 Q2 SOCIAL SCIENCES, INTERDISCIPLINARY

Evaluation and Program Planning Pub Date : 2025-06-16 DOI:10.1016/j.evalprogplan.2025.102638

Mónica Hernández-Campos , Jorge Esteban Prado-Calderón , Antonio Gonzalez-Torres , Francisco José García-Peñalvo

{"title":"Direct measurement of learning outcomes in higher education: A proposal of nine standardized scales for continuous improvement in engineering programs","authors":"Mónica Hernández-Campos , Jorge Esteban Prado-Calderón , Antonio Gonzalez-Torres , Francisco José García-Peñalvo","doi":"10.1016/j.evalprogplan.2025.102638","DOIUrl":null,"url":null,"abstract":"<div><div>Despite the growing interest in measuring learning outcomes as evidence of comprehensive education in higher education institutions, objectively assessing these outcomes remains a challenge. This study addresses this problem by developing and validating nine standardized direct assessment scales tailored for engineering programs, with the goal of enhancing quality assurance. Study 1 focused on designing indicators for the scales through expert workshops and peer judgment processes, looking for content validity using Fleiss Kappa. Study 2 involved instructors using these instruments to assess various learning outcomes in a STEM-focused university. Analysis of 1420 assessments revealed strong validity and reliability indicators, including unifactorial structures, high reliability, discrimination, and appropriate difficulty levels for all scales. The research contributes evidence supporting the content validity of these assessment instruments, offering valuable tools for researchers and practitioners in engineering education to support continuous improvement. These scales have the potential to support future comparative research, inform quality assurance decisions, and enhance comprehensive education practices. Finally, the study discusses its limitations and provides directions for future research.</div></div>","PeriodicalId":48046,"journal":{"name":"Evaluation and Program Planning","volume":"112 ","pages":"Article 102638"},"PeriodicalIF":2.0000,"publicationDate":"2025-06-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Evaluation and Program Planning","FirstCategoryId":"90","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0149718925001053","RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"SOCIAL SCIENCES, INTERDISCIPLINARY","Score":null,"Total":0}

引用次数: 0

Abstract

Despite the growing interest in measuring learning outcomes as evidence of comprehensive education in higher education institutions, objectively assessing these outcomes remains a challenge. This study addresses this problem by developing and validating nine standardized direct assessment scales tailored for engineering programs, with the goal of enhancing quality assurance. Study 1 focused on designing indicators for the scales through expert workshops and peer judgment processes, looking for content validity using Fleiss Kappa. Study 2 involved instructors using these instruments to assess various learning outcomes in a STEM-focused university. Analysis of 1420 assessments revealed strong validity and reliability indicators, including unifactorial structures, high reliability, discrimination, and appropriate difficulty levels for all scales. The research contributes evidence supporting the content validity of these assessment instruments, offering valuable tools for researchers and practitioners in engineering education to support continuous improvement. These scales have the potential to support future comparative research, inform quality assurance decisions, and enhance comprehensive education practices. Finally, the study discusses its limitations and provides directions for future research.

查看原文本刊更多论文

高等教育学习成果的直接测量：工程项目持续改进的九种标准化尺度的建议

尽管人们对衡量学习成果作为高等教育机构全面教育的证据越来越感兴趣，但客观评估这些成果仍然是一个挑战。本研究通过开发和验证为工程项目量身定制的9个标准化的直接评估量表来解决这个问题，其目标是增强质量保证。研究1侧重于通过专家研讨会和同伴判断过程设计量表的指标，并使用Fleiss Kappa寻找内容效度。研究2涉及教师使用这些工具来评估stem大学的各种学习成果。对1420份评估的分析显示了较强的效度和信度指标，包括单因子结构、高信度、可判别性和适用于所有量表的适当难度。该研究为这些评估工具的内容有效性提供了证据，为工程教育的研究人员和实践者提供了有价值的工具，以支持持续改进。这些量表有可能支持未来的比较研究，为质量保证决策提供信息，并加强综合教育实践。最后，讨论了本研究的局限性，并提出了未来研究的方向。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Evaluation and Program Planning SOCIAL SCIENCES, INTERDISCIPLINARY-

CiteScore

3.10

自引率

6.20%

发文量

112

期刊介绍： Evaluation and Program Planning is based on the principle that the techniques and methods of evaluation and planning transcend the boundaries of specific fields and that relevant contributions to these areas come from people representing many different positions, intellectual traditions, and interests. In order to further the development of evaluation and planning, we publish articles from the private and public sectors in a wide range of areas: organizational development and behavior, training, planning, human resource development, health and mental, social services, mental retardation, corrections, substance abuse, and education.