Isabel Hilliger, Constanza Miranda, Gregory Schuit, Fernando Duarte, Martin Anselmo, Denis Parra
{"title":"Evaluating a Learning Analytics Dashboard to Visualize Student Self-Reports of Time-on-task: A Case Study in a Latin American University","authors":"Isabel Hilliger, Constanza Miranda, Gregory Schuit, Fernando Duarte, Martin Anselmo, Denis Parra","doi":"10.1145/3448139.3448203","DOIUrl":"https://doi.org/10.1145/3448139.3448203","url":null,"abstract":"In recent years, instructional design has become even more challenging for teaching staff members in higher education institutions. If instructional design causes student overload, it could lead to superficial learning and decreased student well-being. A strategy to avoid overload is reflecting upon the effectiveness of teaching practices in terms of time-on-task. This article presents a Work-In-Progress conducted to provide teachers with a dashboard to visualize student self-reports of time-on-task regarding subject activities. A questionnaire was applied to 15 instructors during a set trial period to evaluate the perceived usability and usefulness of the dashboard. Preliminary findings reveal that the dashboard helped instructors became aware about the number of hours spent outside of class time. Furthermore, data visualizations of time-on-task evidence enabled them to redesign subject activities. Currently, the dashboard has been adopted by 106 engineering instructors. Future work involves the development of a framework to incorporate user-based improvements.","PeriodicalId":108600,"journal":{"name":"LAK21: 11th International Learning Analytics and Knowledge Conference","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114990050","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"The impact of social norms on students’ online learning behavior: Insights from two randomized controlled trials","authors":"S. Günther","doi":"10.1145/3448139.3448141","DOIUrl":"https://doi.org/10.1145/3448139.3448141","url":null,"abstract":"The provision of comparative feedback is a promising approach in digital learning environments to support learners’ self-regulated learning. Yet, empirical evidence suggests that such feedback can sometimes backfire or may only help learners with relatively high self-regulated learning skills, potentially exacerbating educational inequality. In this paper, we try to overcome such drawbacks by re-evaluating a feedback system based on the social norms theory that has previously led to intriguing results: A social comparison component embedded into the learning platform of a blended learning course (elective module, 58 participants) considerably encouraged online learning during the semester. Moreover, there was no heterogeneity in the behavioral response, suggesting that all subgroups responded similarly to the feedback. To further shed light on the generalizability of these results, this paper presents a follow-up study. Specifically, we conducted a second experiment during the COVID-19 pandemic with a different university course (compulsory module, 118 participants) and a non-overlapping sample and find similar results. The feedback shifted students’ online learning from the end towards the middle of the semester. Overall, the findings suggest that our feedback system has a large impact on students’ online learning and that this desirable impact is present in all subgroup analyses.","PeriodicalId":108600,"journal":{"name":"LAK21: 11th International Learning Analytics and Knowledge Conference","volume":"27 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114269699","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Modeling Consistency Using Engagement Patterns in Online Courses","authors":"Jianing Zhou, Suma Bhat","doi":"10.1145/3448139.3448161","DOIUrl":"https://doi.org/10.1145/3448139.3448161","url":null,"abstract":"Consistency of learning behaviors is known to play an important role in learners’ engagement in a course and impact their learning outcomes. Despite significant advances in the area of learning analytics (LA) in measuring various self-regulated learning behaviors, using LA to measure consistency of online course engagement patterns remains largely unexplored. This study focuses on modeling consistency of learners in online courses to address this research gap. Toward this, we propose a novel unsupervised algorithm that combines sequence pattern mining and ideas from information retrieval with a clustering algorithm to first extract engagement patterns of learners, represent learners in a vector space of these patterns and finally group them into groups with similar consistency levels. Using clickstream data recorded in a popular learning management system over two offerings of a STEM course, we validate our proposed approach to detect learners that are inconsistent in their behaviors. We find that our method not only groups learners by consistency levels, but also provides reliable instructor support at an early stage in a course.","PeriodicalId":108600,"journal":{"name":"LAK21: 11th International Learning Analytics and Knowledge Conference","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116626245","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
I. Jivet, Jacqueline Wong, Maren Scheffel, Manuel Valle Torre, M. Specht, H. Drachsler
{"title":"Quantum of Choice: How learners’ feedback monitoring decisions, goals and self-regulated learning skills are related","authors":"I. Jivet, Jacqueline Wong, Maren Scheffel, Manuel Valle Torre, M. Specht, H. Drachsler","doi":"10.1145/3448139.3448179","DOIUrl":"https://doi.org/10.1145/3448139.3448179","url":null,"abstract":"Learning analytics dashboards (LADs) are designed as feedback tools for learners, but until recently, learners rarely have had a say in how LADs are designed and what information they receive through LADs. To overcome this shortcoming, we have developed a customisable LAD for Coursera MOOCs on which learners can set goals and choose indicators to monitor. Following a mixed-methods approach, we analyse 401 learners’ indicator selection behaviour in order to understand the decisions they make on the LAD and whether learner goals and self-regulated learning skills influence these decisions. We found that learners overwhelmingly chose indicators about completed activities. Goals are not associated with indicator selection behaviour, while help-seeking skills predict learners’ choice of monitoring their engagement in discussions and time management skills predict learners’ interest in procrastination indicators. The findings have implications for our understanding of learners’ use of LADs and their design.","PeriodicalId":108600,"journal":{"name":"LAK21: 11th International Learning Analytics and Knowledge Conference","volume":"26 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123823352","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Lisa-Angelique Lim, D. Gašević, W. Matcha, Nora'ayu Ahmad Uzir, S. Dawson
{"title":"Impact of learning analytics feedback on self-regulated learning: Triangulating behavioural logs with students’ recall","authors":"Lisa-Angelique Lim, D. Gašević, W. Matcha, Nora'ayu Ahmad Uzir, S. Dawson","doi":"10.1145/3448139.3448174","DOIUrl":"https://doi.org/10.1145/3448139.3448174","url":null,"abstract":"Learning analytics (LA) has been presented as a viable solution for scaling timely and personalised feedback to support students’ self-regulated learning (SRL). Research is emerging that shows some positive associations between personalised feedback with students’ learning tactics and strategies as well as time management strategies, both important aspects of SRL. However, the definitive role of feedback on students’ SRL adaptations is under-researched; this requires an examination of students’ recalled experiences with their personalised feedback. Furthermore, an important consideration in feedback impact is the course context, comprised of the learning design and delivery modality. This mixed-methods study triangulates learner trace data from two different course contexts, with students’ qualitative data collected from focus group discussions, to more fully understand the impact of their personalised feedback and to explicate the role of this feedback on students’ SRL adaptations. The quantitative analysis showed the contextualised impact of the feedback on students’ learning and time management strategies in the different courses, while the qualitative analysis highlighted specific ways in which students used their feedback to adjust these and other SRL processes.","PeriodicalId":108600,"journal":{"name":"LAK21: 11th International Learning Analytics and Knowledge Conference","volume":"55 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123392557","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Hyunbin Loh, Dongmin Shin, Seewoo Lee, Jineon Baek, Chanyou Hwang, Youngnam Lee, Yeongmin Cha, S. Kwon, Juneyoung Park, Youngduck Choi
{"title":"Recommendation for Effective Standardized Exam Preparation","authors":"Hyunbin Loh, Dongmin Shin, Seewoo Lee, Jineon Baek, Chanyou Hwang, Youngnam Lee, Yeongmin Cha, S. Kwon, Juneyoung Park, Youngduck Choi","doi":"10.1145/3448139.3448177","DOIUrl":"https://doi.org/10.1145/3448139.3448177","url":null,"abstract":"Finding an optimal learning trajectory is an important question in educational systems. Existing Artificial Intelligence in Education (AiEd) technologies mostly used indirect methods to make the learning process efficient such as recommending contents based on difficulty adjustment, weakness analysis, learning theory, psychometric analysis, or domain specific rules. In this study, we propose a recommender system that optimizes the learning trajectory of a student preparing for a standardized exam by recommending the learning content(question) which directly maximizes the expected score after the consumption of the content. In particular, the proposed RCES model computes the expected score of a user by effectively capturing educational effects. To validate the proposed model in an end-to-end system, we conduct an A/B test on 1713 real students by deploying 4 recommenders to a real mobile application. Result shows that RCES has better educational efficiencies than traditional methods such as expert designed models and item response theory based models.","PeriodicalId":108600,"journal":{"name":"LAK21: 11th International Learning Analytics and Knowledge Conference","volume":"121 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134578504","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Variational Deep Knowledge Tracing for Language Learning","authors":"S. Ruan, Wei Wei, J. Landay","doi":"10.1145/3448139.3448170","DOIUrl":"https://doi.org/10.1145/3448139.3448170","url":null,"abstract":"Deep Knowledge Tracing (DKT), which traces a student’s knowledge change using deep recurrent neural networks, is widely adopted in student cognitive modeling. Current DKT models only predict a student’s performance based on the observed learning history. However, a student’s learning processes often contain latent events not directly observable in the learning history, such as partial understanding, making slips, and guessing answers. Current DKT models fail to model this kind of stochasticity in the learning process. To address this issue, we propose Variational Deep Knowledge Tracing (VDKT), a latent variable DKT model that incorporates stochasticity into DKT through latent variables. We show that VDKT outperforms both a sequence-to-sequence DKT baseline and previous SoTA methods on MAE, F1, and AUC by evaluating our approach on two Duolingo language learning datasets. We also draw various interpretable analyses from VDKT and offer insights into students’ stochastic behaviors in language learning.","PeriodicalId":108600,"journal":{"name":"LAK21: 11th International Learning Analytics and Knowledge Conference","volume":"114 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124788990","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Nicole Gosch, David Andrews, Carla Barreiros, P. Leitner, Elisabeth Staudegger, Martin Ebner, Stefanie N. Lindstaedt
{"title":"Learning Analytics as a Service for Empowered Learners: From Data Subjects to Controllers","authors":"Nicole Gosch, David Andrews, Carla Barreiros, P. Leitner, Elisabeth Staudegger, Martin Ebner, Stefanie N. Lindstaedt","doi":"10.1145/3448139.3448186","DOIUrl":"https://doi.org/10.1145/3448139.3448186","url":null,"abstract":"As Learning Analytics (LA) in the higher education setting increasingly transitions from a field of research to an implemented matter of fact of the learner's experience, the demand of practical guidelines to support its development is rising. LA Policies bring together different perspectives, like the ethical and legal dimensions, into frameworks to guide the way. Usually the first time learners get in touch with LA is at the act of consenting to the LA tool. Utilising an ethical (TRUESSEC) and a legal framework (GDPR), we question whether sincere consent is possible in the higher education setting. Drawing upon this premise, we then show how it might be possible to recognise the autonomy of the learner by providing LA as a service, rather than an intervention. This could indicate a paradigm shift towards the learner as empowered demander. At last, we show how this might be incorporated within the GDPR by also recognising the demand of the higher education institutions to use the learner's data at the same time. These considerations will in the future influence the development of our own LA policy: a LA criteria catalogue.","PeriodicalId":108600,"journal":{"name":"LAK21: 11th International Learning Analytics and Knowledge Conference","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131091677","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
John Saint, Yizhou Fan, Shaveen Singh, D. Gašević, A. Pardo
{"title":"Using process mining to analyse self-regulated learning: a systematic analysis of four algorithms","authors":"John Saint, Yizhou Fan, Shaveen Singh, D. Gašević, A. Pardo","doi":"10.1145/3448139.3448171","DOIUrl":"https://doi.org/10.1145/3448139.3448171","url":null,"abstract":"The conceptualisation of self-regulated learning (SRL) as a process that unfolds over time has influenced the way in which researchers approach analysis. This gave rise to the use of process mining in contemporary SRL research to analyse data about temporal and sequential relations of processes that occur in SRL. However, little attention has been paid to the choice and combinations of process mining algorithms to achieve the nuanced needs of SRL research. We present a study that 1) analysed four process mining algorithms that are most commonly used in the SRL literature – Inductive Miner, Heuristics Miner, Fuzzy Miner, and pMineR; and 2) examined how the metrics produced by the four algorithms complement each. The study looked at micro-level processes that were extracted from trace data collected in an undergraduate course (N=726). The study found that Fuzzy Miner and pMineR offered better insights into SRL than the other two algorithms. The study also found that a combination of metrics produced by several algorithms improved interpretation of temporal and sequential relations between SRL processes. Thus, it is recommended that future studies of SRL combine the use of process mining algorithms and work on new tools and algorithms specifically created for SRL research.","PeriodicalId":108600,"journal":{"name":"LAK21: 11th International Learning Analytics and Knowledge Conference","volume":"12 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130456747","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Namrata Srivastava, Sadia Nawaz, Joshua Newn, J. Lodge, Eduardo Velloso, Sarah M. Erfani, D. Gašević, J. Bailey
{"title":"Are you with me? Measurement of Learners’ Video-Watching Attention with Eye Tracking","authors":"Namrata Srivastava, Sadia Nawaz, Joshua Newn, J. Lodge, Eduardo Velloso, Sarah M. Erfani, D. Gašević, J. Bailey","doi":"10.1145/3448139.3448148","DOIUrl":"https://doi.org/10.1145/3448139.3448148","url":null,"abstract":"Video has become an essential medium for learning. However, there are challenges when using traditional methods to measure how learners attend to lecture videos in video learning analytics, such as difficulty in capturing learners’ attention at a fine-grained level. Therefore, in this paper, we propose a gaze-based metric—“with-me-ness direction” that can measure how learners’ gaze-direction changes when they listen to the instructor’s dialogues in a video-lecture. We analyze the gaze data of 45 participants as they watched a video lecture and measured both the sequences of with-me-ness direction and proportion of time a participant spent looking in each direction throughout the lecture at different levels. We found that although the majority of the time participants followed the instructor’s dialogues, their behaviour of looking-ahead, looking-behind or looking-outside differed by their prior knowledge. These findings open the possibility of using eye-tracking to measure learners’ video-watching attention patterns and examine factors that can influence their attention, thereby helping instructors to design effective learning materials.","PeriodicalId":108600,"journal":{"name":"LAK21: 11th International Learning Analytics and Knowledge Conference","volume":"62 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129173077","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}