Amal Ben Soussia, Chahrazed Labba, A. Roussanaly, A. Boyer
{"title":"Time-dependent metrics to assess performance prediction systems","authors":"Amal Ben Soussia, Chahrazed Labba, A. Roussanaly, A. Boyer","doi":"10.1108/ijilt-07-2022-0149","DOIUrl":null,"url":null,"abstract":"PurposeThe goal is to assess performance prediction systems (PPS) that are used to assist at-risk learners.Design/methodology/approachThe authors propose time-dependent metrics including earliness and stability. The authors investigate the relationships between the various temporal metrics and the precision metrics in order to identify the key earliness points in the prediction process. Authors propose an algorithm for computing earliness. Furthermore, the authors propose using an earliness-stability score (ESS) to investigate the relationship between the earliness of a classifier and its stability. The ESS is used to examine the trade-off between only time-dependent metrics. The aim is to compare its use to the earliness-accuracy score (EAS).FindingsStability and accuracy are proportional when the system's accuracy increases or decreases over time. However, when the accuracy stagnates or varies slightly, the system's stability is decreasing rather than stagnating. As a result, the use of ESS and EAS is complementary and allows for a better definition of the point of earliness in time by studying the relation-ship between earliness and accuracy on the one hand and earliness and stability on the other.Originality/valueWhen evaluating the performance of PPS, the temporal dimension is an important factor that is overlooked by traditional measures current metrics are not well suited to assessing PPS’s ability to predict correctly at the earliest, as well as monitoring predictions stability and evolution over time. Thus, in this work, the authors propose time-dependent metrics, including earliness, stability and the trade-offs, with objective to assess PPS over time.","PeriodicalId":51872,"journal":{"name":"International Journal of Information and Learning Technology","volume":null,"pages":null},"PeriodicalIF":2.4000,"publicationDate":"2022-10-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"International Journal of Information and Learning Technology","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1108/ijilt-07-2022-0149","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS","Score":null,"Total":0}
引用次数: 0
Abstract
PurposeThe goal is to assess performance prediction systems (PPS) that are used to assist at-risk learners.Design/methodology/approachThe authors propose time-dependent metrics including earliness and stability. The authors investigate the relationships between the various temporal metrics and the precision metrics in order to identify the key earliness points in the prediction process. Authors propose an algorithm for computing earliness. Furthermore, the authors propose using an earliness-stability score (ESS) to investigate the relationship between the earliness of a classifier and its stability. The ESS is used to examine the trade-off between only time-dependent metrics. The aim is to compare its use to the earliness-accuracy score (EAS).FindingsStability and accuracy are proportional when the system's accuracy increases or decreases over time. However, when the accuracy stagnates or varies slightly, the system's stability is decreasing rather than stagnating. As a result, the use of ESS and EAS is complementary and allows for a better definition of the point of earliness in time by studying the relation-ship between earliness and accuracy on the one hand and earliness and stability on the other.Originality/valueWhen evaluating the performance of PPS, the temporal dimension is an important factor that is overlooked by traditional measures current metrics are not well suited to assessing PPS’s ability to predict correctly at the earliest, as well as monitoring predictions stability and evolution over time. Thus, in this work, the authors propose time-dependent metrics, including earliness, stability and the trade-offs, with objective to assess PPS over time.
期刊介绍:
International Journal of Information and Learning Technology (IJILT) provides a forum for the sharing of the latest theories, applications, and services related to planning, developing, managing, using, and evaluating information technologies in administrative, academic, and library computing, as well as other educational technologies. Submissions can include research: -Illustrating and critiquing educational technologies -New uses of technology in education -Issue-or results-focused case studies detailing examples of technology applications in higher education -In-depth analyses of the latest theories, applications and services in the field The journal provides wide-ranging and independent coverage of the management, use and integration of information resources and learning technologies.