Time-dependent metrics to assess performance prediction systems

IF 2.4 Q3 COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS
Amal Ben Soussia, Chahrazed Labba, A. Roussanaly, A. Boyer
{"title":"Time-dependent metrics to assess performance prediction systems","authors":"Amal Ben Soussia, Chahrazed Labba, A. Roussanaly, A. Boyer","doi":"10.1108/ijilt-07-2022-0149","DOIUrl":null,"url":null,"abstract":"PurposeThe goal is to assess performance prediction systems (PPS) that are used to assist at-risk learners.Design/methodology/approachThe authors propose time-dependent metrics including earliness and stability. The authors investigate the relationships between the various temporal metrics and the precision metrics in order to identify the key earliness points in the prediction process. Authors propose an algorithm for computing earliness. Furthermore, the authors propose using an earliness-stability score (ESS) to investigate the relationship between the earliness of a classifier and its stability. The ESS is used to examine the trade-off between only time-dependent metrics. The aim is to compare its use to the earliness-accuracy score (EAS).FindingsStability and accuracy are proportional when the system's accuracy increases or decreases over time. However, when the accuracy stagnates or varies slightly, the system's stability is decreasing rather than stagnating. As a result, the use of ESS and EAS is complementary and allows for a better definition of the point of earliness in time by studying the relation-ship between earliness and accuracy on the one hand and earliness and stability on the other.Originality/valueWhen evaluating the performance of PPS, the temporal dimension is an important factor that is overlooked by traditional measures current metrics are not well suited to assessing PPS’s ability to predict correctly at the earliest, as well as monitoring predictions stability and evolution over time. Thus, in this work, the authors propose time-dependent metrics, including earliness, stability and the trade-offs, with objective to assess PPS over time.","PeriodicalId":51872,"journal":{"name":"International Journal of Information and Learning Technology","volume":" ","pages":""},"PeriodicalIF":2.4000,"publicationDate":"2022-10-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"International Journal of Information and Learning Technology","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1108/ijilt-07-2022-0149","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS","Score":null,"Total":0}
引用次数: 0

Abstract

PurposeThe goal is to assess performance prediction systems (PPS) that are used to assist at-risk learners.Design/methodology/approachThe authors propose time-dependent metrics including earliness and stability. The authors investigate the relationships between the various temporal metrics and the precision metrics in order to identify the key earliness points in the prediction process. Authors propose an algorithm for computing earliness. Furthermore, the authors propose using an earliness-stability score (ESS) to investigate the relationship between the earliness of a classifier and its stability. The ESS is used to examine the trade-off between only time-dependent metrics. The aim is to compare its use to the earliness-accuracy score (EAS).FindingsStability and accuracy are proportional when the system's accuracy increases or decreases over time. However, when the accuracy stagnates or varies slightly, the system's stability is decreasing rather than stagnating. As a result, the use of ESS and EAS is complementary and allows for a better definition of the point of earliness in time by studying the relation-ship between earliness and accuracy on the one hand and earliness and stability on the other.Originality/valueWhen evaluating the performance of PPS, the temporal dimension is an important factor that is overlooked by traditional measures current metrics are not well suited to assessing PPS’s ability to predict correctly at the earliest, as well as monitoring predictions stability and evolution over time. Thus, in this work, the authors propose time-dependent metrics, including earliness, stability and the trade-offs, with objective to assess PPS over time.
用于评估性能预测系统的与时间相关的指标
目的:目的是评估用于帮助有风险的学习者的表现预测系统(PPS)。作者提出了与时间相关的指标,包括早期性和稳定性。作者研究了各种时间指标与精度指标之间的关系,以确定预测过程中的关键早期点。作者提出了一种计算早期的算法。此外,作者提出使用早期稳定性评分(ESS)来研究分类器的早期性与其稳定性之间的关系。ESS仅用于检查与时间相关的度量之间的权衡。目的是将其与早期准确性评分(EAS)进行比较。当系统的精度随时间增加或减少时,稳定性和精度成正比。然而,当精度停滞或略有变化时,系统的稳定性不是停滞而是下降。因此,ESS和EAS的使用是互补的,通过研究早期和准确性以及早期和稳定性之间的关系,可以更好地定义时间的早期点。原创性/价值在评估PPS的表现时,时间维度是一个被传统测量方法所忽视的重要因素,目前的指标并不适合于评估PPS最早正确预测的能力,也不适合监测预测的稳定性和随时间的演变。因此,在这项工作中,作者提出了与时间相关的指标,包括早期性、稳定性和权衡,目的是随着时间的推移评估PPS。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
International Journal of Information and Learning Technology
International Journal of Information and Learning Technology COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS-
CiteScore
6.10
自引率
3.30%
发文量
33
期刊介绍: International Journal of Information and Learning Technology (IJILT) provides a forum for the sharing of the latest theories, applications, and services related to planning, developing, managing, using, and evaluating information technologies in administrative, academic, and library computing, as well as other educational technologies. Submissions can include research: -Illustrating and critiquing educational technologies -New uses of technology in education -Issue-or results-focused case studies detailing examples of technology applications in higher education -In-depth analyses of the latest theories, applications and services in the field The journal provides wide-ranging and independent coverage of the management, use and integration of information resources and learning technologies.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信