LAK23: 13th International Learning Analytics and Knowledge Conference最新文献

{"title":"An Integrated Model of Feedback and Assessment: From fine grained to holistic programmatic review","authors":"S. Dawson, Abelardo Pardo, Fatemeh Salehian Kia, E. Panadero","doi":"10.1145/3576050.3576074","DOIUrl":"https://doi.org/10.1145/3576050.3576074","url":null,"abstract":"Abstract: Research in learning analytics (LA) has long held a strong interest in improving student self-regulated learning and measuring the impact of feedback on student outcomes. Despite more than a decade of work in this space very little is known around the contextual factors that influence the topics and diversity of feedback and assessment a student encounters during their full program of study. This paper presents research investigating the institutional adoption of a personalized feedback tool. The reported findings illustrate an association between the topics of feedback, student performance, year level of the course and discipline. The results highlight the need for LA research to capture feedback, assessment and learning outcomes over an entire program of study. Herein we propose a more integrated model drawing on contemporary understandings of feedback with current research findings. The goal is to push LA towards addressing more complex teaching and learning processes from a systems lens. The model posed in this paper begins to illustrate where and how LA can address noted deficits in education practice to better understand how feedback and assessment are enacted by instructors and interpreted by students.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"28 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126183516","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Are They Learning or Guessing? Investigating Trial-and-Error Behavior with Limited Test Attempts","authors":"Bowen Hui","doi":"10.1145/3576050.3576068","DOIUrl":"https://doi.org/10.1145/3576050.3576068","url":null,"abstract":"Mastery learning and deliberate practice promote personalized learning, allowing the learner to improve through a repetitive and targeted approach. Unfortunately, this pedagogy is challenging to implement in classrooms where everyone is expected to learn at the same pace. Various studies have successfully demonstrated that certain aspects of mastery learning can be integrated into the curriculum. We adopt a similar pedagogical strategy and explain our implementation approach with online assessments. Since this is a new pedagogical approach to assessing student learning, we collected data to investigate test-taking behavior and evaluated potential learning gains in this new test format. As part of this endeavor, we developed a model to detect trial-and-error sequences in test attempts. Our results point to a small percentage of guessing behavior, which is encouraging evidence supporting this test approach is a viable way to implement mastery learning in our curriculum.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"20 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128209348","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Identifying Gaze Behavior Evolution via Temporal Fully-Weighted Scanpath Graphs","authors":"Eduardo Davalos, Caleb Vatral, Clayton Cohn, Joyce Horn Fonteles, G. Biswas, Naveeduddin Mohammed, Madison Lee, Daniel Levin","doi":"10.1145/3576050.3576117","DOIUrl":"https://doi.org/10.1145/3576050.3576117","url":null,"abstract":"Eye-tracking technology has expanded our ability to quantitatively measure human perception. This rich data source has been widely used to characterize human behavior and cognition. However, eye-tracking analysis has been limited in its applicability, as contextualizing gaze to environmental artifacts is non-trivial. Moreover, the temporal evolution of gaze behavior through open-ended environments where learners are alternating between tasks often remains unclear. In this paper, we propose temporal fully-weighted scanpath graphs as a novel representation of gaze behavior and combine it with a clustering scheme to obtain high-level gaze summaries that can be mapped to cognitive tasks via network metrics and cluster mean graphs. In a case study with nurse simulation-based team training, our approach was able to explain changes in gaze behavior with respect to key events during the simulation. By identifying cognitive tasks via gaze behavior, learners’ strategies can be evaluated to create online performance metrics and personalized feedback.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"81 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127277954","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Lost in Translation: Determining the Generalizability of Temporal Models across Course Contexts ","authors":"Joshua D. Quick, Benjamin A. Motz, Anastasia S. Morrone","doi":"10.1145/3576050.3576092","DOIUrl":"https://doi.org/10.1145/3576050.3576092","url":null,"abstract":"A common activity in learning analytics research is to demonstrate a new analytical technique by applying it to data from a single course. We explore whether the value of an analytical approach might generalize across course contexts. Accordingly, we conduct a conceptual replication of a well-cited temporal modeling study using self-regulated learning (SRL) taxonomies. We attempt to conceptually replicate this previous work through the analysis of 411 students across 19 courses’ trace event data. Using established SRL categorizations, learner actions are sequenced to identify regular clusters of interaction through hierarchical clustering methods. These clusters are then compared with the entire data corpus and each other through the development of first-order Markov models to develop process maps. Our findings indicate that, although some general patterns of SRL can generalize, these results are more limited at higher scales. Comparing these clusters of interaction along students’ performance in courses also indicates some relationships between activity and outcomes, though this finding is also limited in relation to the complexity introduced by scaling out these methods. We discuss how these temporal models should be viewed when making descriptive and qualitative inferences about students’ activity in digital learning environments.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"24 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129156540","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Using submission log data to investigate novice programmers’ employment of debugging strategies","authors":"Qianhui Liu, L. Paquette","doi":"10.1145/3576050.3576094","DOIUrl":"https://doi.org/10.1145/3576050.3576094","url":null,"abstract":"Debugging is a distinct subject in programming that is both comprehensive and challenging for novice programmers. However, instructors have limited opportunities to gain insights into the difficulties students encountered in isolated debugging processes. While qualitative studies have identified debugging strategies that novice programmers use and how they relate to theoretical debugging frameworks, limited larger scale quantitative analyses have been conducted to investigate how students’ debugging behaviors observed in log data align with the identified strategies and how they relate to successful debugging. In this study, we used submission log data to understand how the existing debugging strategies are employed by students in an introductory CS course when solving homework problems. We identified strategies from existing debugging literature that can be observed with trace data and extracted features to reveal how efficient debugging is associated with debugging strategy usage. Our findings both align with and contradict past assumptions from previous studies by suggesting that minor code edition can be a beneficial strategy and that width and depth aggregations of the same debugging behavior can reveal opposite effects on debugging efficiency.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"29 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115789025","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Using Teacher Dashboards to Customize Lesson Plans for a Problem-Based, Middle School STEM Curriculum","authors":"Nicole M. Hutchins, G. Biswas","doi":"10.1145/3576050.3576100","DOIUrl":"https://doi.org/10.1145/3576050.3576100","url":null,"abstract":"Keeping K-12 teachers engaged during students’ learning and problem solving in technology-enhanced, integrated problem-based learning (PBL) has been shown to support deeper student involvement, and, therefore, better success learning difficult science, computing, and engineering concepts and practices. However, students’ learning processes and corresponding difficulties are not easily noticed by teachers as students learn from these environments as processes are captured through mouse clicks, drag and drop actions, and other low-level activities. As such, teachers find it difficult to set up meaningful interactions with students while also maintaining the focus on student-centered learning. Little research has examined dashboard-supported responsive teaching practices for K-12 PBL. This study examined 8 teachers as they used a co-designed teacher dashboard to assess and respond to students’ learning and strategies during an integrated, PBL STEM curriculum. Teachers completed a series of 5 “planning period simulations” leveraging the dashboard and think-aloud protocols were implemented, supported by semi-structured interview questions, to enable the teachers to verbalize their thought and evaluation processes. Content analysis and epistemic network analysis were conducted to analyze the simulations. Understanding how teachers use dashboards to support evidence-based teaching practices during technology-enhanced curricula is critical for improving teacher support and preparation.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"92 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121225798","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Multimodal Predictive Student Modeling with Multi-Task Transfer Learning","authors":"Andrew Emerson, Wookhee Min, Jonathan P. Rowe, R. Azevedo, James C. Lester","doi":"10.1145/3576050.3576101","DOIUrl":"https://doi.org/10.1145/3576050.3576101","url":null,"abstract":"Game-based learning environments have the distinctive capacity to promote learning experiences that are both engaging and effective. Recent advances in sensor-based technologies (e.g., facial expression analysis and eye gaze tracking) and natural language processing have introduced the opportunity to leverage multimodal data streams for learning analytics. Learning analytics and student modeling informed by multimodal data captured during students’ interactions with game-based learning environments hold significant promise for designing effective learning environments that detect unproductive student behaviors and provide adaptive support for students during learning. Learning analytics frameworks that can accurately predict student learning outcomes early in students’ interactions hold considerable promise for enabling environments to dynamically adapt to individual student needs. In this paper, we investigate a multimodal, multi-task predictive student modeling framework for game-based learning environments. The framework is evaluated on two datasets of game-based learning interactions from two student populations (n=61 and n=118) who interacted with two versions of a game-based learning environment for microbiology education. The framework leverages available multimodal data channels from the datasets to simultaneously predict student post-test performance and interest. In addition to inducing models for each dataset individually, this work investigates the ability to use information learned from one source dataset to improve models based on another target dataset (i.e., transfer learning using pre-trained models). Results from a series of ablation experiments indicate the differences in predictive capacity among a combination of modalities including gameplay, eye gaze, facial expressions, and reflection text for predicting the two target variables. In addition, multi-task models were able to improve predictive performance compared to single-task baselines for one target variable, but not both. Lastly, transfer learning showed promise in improving predictive capacity in both datasets.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"269 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116543545","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Embracing Trustworthiness and Authenticity in the Validation of Learning Analytics Systems","authors":"Max van Haastrecht, Matthieu J. S. Brinkhuis, Jessica Peichl, Bernd Remmele, Marco Spruit","doi":"10.1145/3576050.3576060","DOIUrl":"https://doi.org/10.1145/3576050.3576060","url":null,"abstract":"Learning analytics sits in the middle space between learning theory and data analytics. The inherent diversity of learning analytics manifests itself in an epistemology that strikes a balance between positivism and interpretivism, and knowledge that is sourced from theory and practice. In this paper, we argue that validation approaches for learning analytics systems should be cognisant of these diverse foundations. Through a systematic review of learning analytics validation research, we find that there is currently an over-reliance on positivistic validity criteria. Researchers tend to ignore interpretivistic criteria such as trustworthiness and authenticity. In the 38 papers we analysed, researchers covered positivistic validity criteria 221 times, whereas interpretivistic criteria were mentioned 37 times. We motivate that learning analytics can only move forward with holistic validation strategies that incorporate “thick descriptions” of educational experiences. We conclude by outlining a planned validation study using argument-based validation, which we believe will yield meaningful insights by considering a diverse spectrum of validity criteria.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"255 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116578582","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Using a Webcam Based Eye-tracker to Understand Students’ Thought Patterns and Reading Behaviors in Neurodivergent Classrooms","authors":"Aaron Y. Wong, Richard L. Bryck, R. Baker, Stephen Hutt, Caitlin Mills","doi":"10.1145/3576050.3576115","DOIUrl":"https://doi.org/10.1145/3576050.3576115","url":null,"abstract":"Previous learning analytics efforts have attempted to leverage the link between students’ gaze behaviors and learning experiences to build effective real-time interventions. Historically, however, these technologies have not been scalable due to the high cost of eye-tracking devices. Further, such efforts have been almost exclusively focused on neurotypical students, despite recent work that suggests a “one size fits many” approach can disadvantage neurodivergent students. Here we attempt to address these limitations by examining the validity and applicability of using scalable, webcam-based eye tracking as a basis for adaptively responding to neurodivergent students in an educational setting. Forty-three neurodivergent students read a text and answered questions about their in-situ thought patterns while a webcam-based eye tracker assessed their gaze locations. Results indicate that eye-tracking measures were sensitive to: 1) moments when students experienced difficulty disengaging from their own thoughts and 2) students’ familiarity with the text. Our findings highlight the fact that a free, open-source, webcam-based eye-tracker can be used to assess differences in reading patterns and online thought patterns. We discuss the implications and possible applications of these results, including the idea that webcam-based eye tracking may be a viable solution for designing real-time interventions for neurodivergent student populations.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127654614","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Impact of window size on the generalizability of collaboration quality estimation models developed using Multimodal Learning Analytics","authors":"Pankaj Chejara, L. Prieto, M. Rodríguez-Triana, Adolfo Ruiz-Calleja, M. Khalil","doi":"10.1145/3576050.3576143","DOIUrl":"https://doi.org/10.1145/3576050.3576143","url":null,"abstract":"Multimodal Learning Analytics (MMLA) has been applied to collaborative learning, often to estimate collaboration quality with the use of multimodal data, which often have uneven time scales. The difference in time scales is usually handled by dividing and aggregating data using a fixed-size time window. So far, the current MMLA research lacks a systematic exploration of whether and how much window size affects the generalizability of collaboration quality estimation models. In this paper, we investigate the impact of different window sizes (e.g., 30 seconds, 60s, 90s, 120s, 180s, 240s) on the generalizability of classification models for collaboration quality and its underlying dimensions (e.g., argumentation). Our results from an MMLA study involving the use of audio and log data showed that a 60 seconds window size enabled the development of more generalizable models for collaboration quality (AUC 61%) and argumentation (AUC 64%). In contrast, for modeling dimensions focusing on coordination, interpersonal relationship, and joint information processing, a window size of 180 seconds led to better performance in terms of across-context generalizability (on average from 56% AUC to 63% AUC). These findings have implications for the eventual application of MMLA in authentic practice.","PeriodicalId":394433,"journal":{"name":"LAK23: 13th International Learning Analytics and Knowledge Conference","volume":"51 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131807650","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}