Constructing and validating authentic assessments: the case of a new technology-based assessment of economic literacy

IF 1.6 Q2 EDUCATION & EDUCATIONAL RESEARCH
Nina Charlotte Johanna Welsandt, Fabio Fortunati, Esther Winther, Hermann Josef Abs
{"title":"Constructing and validating authentic assessments: the case of a new technology-based assessment of economic literacy","authors":"Nina Charlotte Johanna Welsandt, Fabio Fortunati, Esther Winther, Hermann Josef Abs","doi":"10.1186/s40461-024-00158-0","DOIUrl":null,"url":null,"abstract":"<h3 data-test=\"abstract-sub-heading\">Background</h3><p>Authentic situations are considered a source of learning due to their real world relevance. This can encourage learners to acquire new knowledge. Increasing digitisation and associated resources, such as professional development opportunities for teachers, technology tools, or digital equipment for schools enable the development and implementation of authentic assessments. The basic academic principles for acquiring economic literacy are already provided in lower secondary school. This article examines, using the example of a new authentic technology-based assessment (TBA)—<i>Economic Literacy—Assessing the Status Quo in Grade 8</i> (ECON 2022) -, the processes involved in constructing a TBA. The purpose is to develop a curricular valid measurement instrument for surveying the current state of economic literacy in the 8th grade of a German federal state. This study explores which economic competencies students—typically between 14 and 15 years of age—possess in Grade 8, and what level of competence can therefore be expected of them at the beginning of a vocational training programme. The assessment is geared toward the curriculum of the subject of economics and is based on a domain model. This article presents the background and construction process for the development of ECON 2022 as a TBA.</p><h3 data-test=\"abstract-sub-heading\">Methods</h3><p>To check the validity of test construction with a focus on the implementation of the authentic assessment and an analysis of difficulty-generating characteristics, the ECON 2022 test items were validated with an expert survey (<i>N</i> = 25). The two-stage data analysis comprised a descriptive quantifying analysis of the rating from the difficulty-generating characteristics specificity, cognitive demand and modelling and the design criterion authenticity. A set of experts rated the criteria. The expert survey was then compared with a previously conducted rating by the research team. The analysis of free-text comments on individual items was carried out discursively and qualitatively by the research team. Both sources of information were used to adapt the test items to measured item difficulties from the field test. For this purpose, items of great difficulty were changed to slightly easier items. In this context, the paper focuses on two central research questions: - How does the authenticity of a test environment relate to difficulty-generating criteria at item level? - Does the authenticity of a test environment have an impact on test results?</p><h3 data-test=\"abstract-sub-heading\">Results</h3><p>Results are as follows. (1) The ECON 2022 assessment offers an example of a test design in which the use of TBAs can support innovative and interactive item development. (2) Using the ECON 2022 assessment enabled the implementation of an assessment of economic literacy using authentic situations and the mapping of different facets of economic competence. (3) The validation study showed that the actual item difficulty did not correlate significantly with the authenticity of the assessment, and authenticity thus did not contribute to item difficulty.</p><h3 data-test=\"abstract-sub-heading\">Conclusion</h3><p>The results of the study show that we were successful in developing an authentic TBA in ECON 2022. ECON 2022 allows us to examine economic literacy in schools with a high degree of curricular validity and relevance and to analyse what level of competence and knowledge can be expected of students when they enter a vocational training occupation.</p>","PeriodicalId":38550,"journal":{"name":"Empirical Research in Vocational Education and Training","volume":null,"pages":null},"PeriodicalIF":1.6000,"publicationDate":"2024-04-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Empirical Research in Vocational Education and Training","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1186/s40461-024-00158-0","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}
引用次数: 0

Abstract

Background

Authentic situations are considered a source of learning due to their real world relevance. This can encourage learners to acquire new knowledge. Increasing digitisation and associated resources, such as professional development opportunities for teachers, technology tools, or digital equipment for schools enable the development and implementation of authentic assessments. The basic academic principles for acquiring economic literacy are already provided in lower secondary school. This article examines, using the example of a new authentic technology-based assessment (TBA)—Economic Literacy—Assessing the Status Quo in Grade 8 (ECON 2022) -, the processes involved in constructing a TBA. The purpose is to develop a curricular valid measurement instrument for surveying the current state of economic literacy in the 8th grade of a German federal state. This study explores which economic competencies students—typically between 14 and 15 years of age—possess in Grade 8, and what level of competence can therefore be expected of them at the beginning of a vocational training programme. The assessment is geared toward the curriculum of the subject of economics and is based on a domain model. This article presents the background and construction process for the development of ECON 2022 as a TBA.

Methods

To check the validity of test construction with a focus on the implementation of the authentic assessment and an analysis of difficulty-generating characteristics, the ECON 2022 test items were validated with an expert survey (N = 25). The two-stage data analysis comprised a descriptive quantifying analysis of the rating from the difficulty-generating characteristics specificity, cognitive demand and modelling and the design criterion authenticity. A set of experts rated the criteria. The expert survey was then compared with a previously conducted rating by the research team. The analysis of free-text comments on individual items was carried out discursively and qualitatively by the research team. Both sources of information were used to adapt the test items to measured item difficulties from the field test. For this purpose, items of great difficulty were changed to slightly easier items. In this context, the paper focuses on two central research questions: - How does the authenticity of a test environment relate to difficulty-generating criteria at item level? - Does the authenticity of a test environment have an impact on test results?

Results

Results are as follows. (1) The ECON 2022 assessment offers an example of a test design in which the use of TBAs can support innovative and interactive item development. (2) Using the ECON 2022 assessment enabled the implementation of an assessment of economic literacy using authentic situations and the mapping of different facets of economic competence. (3) The validation study showed that the actual item difficulty did not correlate significantly with the authenticity of the assessment, and authenticity thus did not contribute to item difficulty.

Conclusion

The results of the study show that we were successful in developing an authentic TBA in ECON 2022. ECON 2022 allows us to examine economic literacy in schools with a high degree of curricular validity and relevance and to analyse what level of competence and knowledge can be expected of students when they enter a vocational training occupation.

Abstract Image

构建和验证真实评估:基于新技术的经济素养评估案例
背景由于真实情境与现实世界息息相关,因此被认为是学习的源泉。这可以鼓励学习者获取新知识。数字化和相关资源的不断增加,如为教师提供的专业发展机会、技术工具或为学校提供的数字设备,都有助于开发和实施真实的评估。初中阶段已经提供了获取经济素养的基本学术原则。本文以一个新的基于技术的真实评估(TBA)--八年级经济素养-现状评估(ECON 2022)--为例,探讨了构建一个基于技术的真实评估的过程。目的是开发一种课程有效的测量工具,用于调查德国联邦州八年级学生的经济素养现状。本研究探讨了八年级学生(通常为 14 至 15 岁)所具备的经济能力,以及在职业培训课程开始时可期待他们达到的能力水平。评估以经济学科的课程为导向,以领域模型为基础。本文介绍了《经济学2022》作为TBA的开发背景和构建过程。方法为了检查测试构建的有效性,重点关注真实评估的实施和难度产生特征的分析,《经济学2022》测试项目通过专家调查(N = 25)进行了验证。两阶段数据分析包括对难度产生特征的具体性、认知需求和建模以及设计标准的真实性进行描述性量化分析。一组专家对标准进行评分。然后将专家调查与研究小组之前进行的评级进行比较。研究小组对个别项目的自由文本评论进行了辨证和定性分析。这两种信息来源都被用来调整测试项目,以适应实地测试中测出的项目难度。为此,难度很大的题目被改成了稍微简单的题目。在这种情况下,本文重点讨论两个核心研究问题:- 测试环境的真实性与项目层面的难度产生标准有什么关系?- 考试环境的真实性是否会对考试结果产生影响?(1) ECON 2022 评估提供了一个使用 TBA 支持创新和互动项目开发的测试设计实例。(2)使用《经济学 2022》评估能够利用真实情境实施经济素养评估,并绘制经济能力的不同方面。(3) 验证研究表明,实际题目难度与测评的真实性没有显著相关性,因此真实性对题目难度没有影响。ECON 2022 使我们能够以高度的课程有效性和相关性来考察学校的经济素养,并分析学生在进入职业培训行业时应具备的能力和知识水平。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
CiteScore
3.40
自引率
7.70%
发文量
9
审稿时长
13 weeks
期刊介绍: The main focus of this journal is to provide a platform for original empirical investigations in the field of professional, vocational and technical education, comparing the effectiveness, efficiency and equity of different vocational education systems at the school, company and systemic level. The journal fills a gap in the existing literature focusing on empirically-oriented academic research and stimulating the interest in strengthening the vocational part of the educational system, both at the basic and higher education level.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信