{"title":"How does Students’ Affect in Virtual Learning Relate to Their Outcomes? A Systematic Review Challenging the Positive-Negative Dichotomy","authors":"Shamya Karumbaiah, R. Baker, Yan Tao, Ziyang Liu","doi":"10.1145/3506860.3506863","DOIUrl":"https://doi.org/10.1145/3506860.3506863","url":null,"abstract":"Several emotional theories that inform the design of Virtual Learning Environments (VLEs) categorize affect as either positive or negative. However, the relationship between affect and learning appears to be more complex than that. Despite several empirical investigations in the last fifteen years, including a few that have attempted to complexify the role of affect in students’ learning in VLE, there has not been an attempt to synthesize the evidence across them. To bridge this gap, we conducted a systematic review of empirical studies that examined the relationship between student outcomes and the affect that arises during their interaction with a VLE. Our synthesis of results across thirty-nine papers suggests that except engagement, all of the commonly studied affective states (confusion, frustration, and boredom) have mixed relationships with outcomes. We further explored the differences in student demographics and study context to explain the variation in the results. Some of our key findings include poorer learning outcomes arising for confusion in classrooms (versus lab studies), differences in brief versus prolonged confusion and resolved versus persistent confusion, more positive (versus null) results for engagement in learning games, and more significant results for rarer affective states like frustration with automated affect detectors (versus student self-reports). We conclude that more careful attention must be paid to contextual differences in affect's role in student learning. We discuss the implication of this review for VLE design and research.","PeriodicalId":185465,"journal":{"name":"LAK22: 12th International Learning Analytics and Knowledge Conference","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134162991","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Towards Multi-modal Evaluation of Eye-tracked Virtual Heritage Environment","authors":"Tzi-Dong Jeremy Ng, Xiao Hu, Y. Que","doi":"10.1145/3506860.3506881","DOIUrl":"https://doi.org/10.1145/3506860.3506881","url":null,"abstract":"In times of pandemic-induced challenges, virtual reality (VR) allows audience to learn about cultural heritage sites without temporal and spatial constraints. The design of VR content is largely determined by professionals, while evaluations of content often rely on learners’ self-report data. Learners’ attentional focus and understanding of VR content might be affected by the presence or absence of different multimedia elements including text and audio-visuals. It remains an open question which design variations are more conducive for learning about heritage sites. Leveraging eye-tracking, a technology often adopted in recent multimodal learning analytics (MmLA) research, we conducted an experiment to collect and analyze 40 learners’ eye movement and self-reported data. Results of statistical tests and heatmap elicitation interviews indicate that 1) text in the VR environment helped learners better understand the presented heritage sites, regardless of having audio narration or not, 2) text diverted learners’ attention away from other visual elements that contextualized the heritage sites, 3) exclusively having audio narration best simulated the experience of a real-world heritage tour, 4) narration accompanying text prompted learners to read the text faster. We make recommendations for improving the design of VR learning materials and discuss the implications for MmLA research.","PeriodicalId":185465,"journal":{"name":"LAK22: 12th International Learning Analytics and Knowledge Conference","volume":"56 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134245531","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Lixiang Yan, Linxuan Zhao, D. Gašević, Roberto Martínez Maldonado
{"title":"Scalability, Sustainability, and Ethicality of Multimodal Learning Analytics","authors":"Lixiang Yan, Linxuan Zhao, D. Gašević, Roberto Martínez Maldonado","doi":"10.1145/3506860.3506862","DOIUrl":"https://doi.org/10.1145/3506860.3506862","url":null,"abstract":"Multimodal Learning Analytics (MMLA) innovations are commonly aimed at supporting learners in physical learning spaces through state-of-the-art sensing technologies and analysis techniques. Although a growing body of MMLA research has demonstrated the potential benefits of sensor-based technologies in education, whether their use can be scalable, sustainable, and ethical remains questionable. Such uncertainty can limit future research and the potential adoption of MMLA by educational stakeholders in authentic learning situations. To address this, we systematically reviewed the methodological, operational, and ethical challenges faced by current MMLA works that can affect the scalability and sustainability of future MMLA innovations. A total of 96 peer-reviewed articles published after 2010 were included. The findings were summarised into three recommendations, including i) improving reporting standards by including sufficient details about sensors, analysis techniques, and the full disclosure of evaluation metrics, ii) fostering interdisciplinary collaborations among experts in learning analytics, software, and hardware engineering to develop affordable sensors and upgrade MMLA innovations that used discontinued technologies, and iii) developing ethical guidelines to address the potential risks of bias, privacy, and equality concerns with using MMLA innovations. Through these future research directions, MMLA can remain relevant and eventually have actual impacts on educational practices.","PeriodicalId":185465,"journal":{"name":"LAK22: 12th International Learning Analytics and Knowledge Conference","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116574120","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Amber J. Dood, Blair A. Winograd, S. Finkenstaedt-Quinn, A. Gere, G. Shultz
{"title":"PeerBERT: Automated Characterization of Peer Review Comments across Courses","authors":"Amber J. Dood, Blair A. Winograd, S. Finkenstaedt-Quinn, A. Gere, G. Shultz","doi":"10.1145/3506860.3506892","DOIUrl":"https://doi.org/10.1145/3506860.3506892","url":null,"abstract":"Writing-to-learn pedagogies are an evidence-based practice known to aid students in constructing knowledge. Barriers exist for the implementation of such assignments; namely, instructors feel they do not have time to provide each student with feedback. To ease implementation of writing-to-learn assignments at scale, we have incorporated automated peer review, which facilitates peer review without input from the instructor. Participating in peer review can positively impact students’ learning and allow students to receive feedback on their writing. Instructors may want to monitor these peer interactions and gain insight into their students’ understanding using the feedback generated by their peers. To facilitate instructors’ use of the content from students’ peer review comments, we pre-trained a transformer model called PeerBERT. PeerBERT was fine-tuned on several downstream tasks to categorize students’ peer review comments as praise, problem/solution, or verification/summary. The model exhibits high accuracy, even across different peer review prompts, assignments, and courses. Additional downstream tasks label problem/solution peer review comments as one or more types: writing/formatting, missing content/needs elaboration, and incorrect content. This approach can help instructors pinpoint common issues in student writing by parsing out which comments are problem/solution and which type of problem/solution students identify.","PeriodicalId":185465,"journal":{"name":"LAK22: 12th International Learning Analytics and Knowledge Conference","volume":"29 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126099277","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
W. Leite, S. Roy, Nilanjana Chakraborty, G. Michailidis, A. Huggins-Manley, S. D’Mello, Mohamad Kazem Shirani Faradonbeh, Emily Jensen, H. Kuang, Zeyuan Jing
{"title":"A novel video recommendation system for algebra: An effectiveness evaluation study","authors":"W. Leite, S. Roy, Nilanjana Chakraborty, G. Michailidis, A. Huggins-Manley, S. D’Mello, Mohamad Kazem Shirani Faradonbeh, Emily Jensen, H. Kuang, Zeyuan Jing","doi":"10.1145/3506860.3506906","DOIUrl":"https://doi.org/10.1145/3506860.3506906","url":null,"abstract":"This study presents a novel video recommendation system for an algebra virtual learning environment (VLE) that leverages ideas and methods from engagement measurement, item response theory, and reinforcement learning. Following Vygotsky's Zone of Proximal Development (ZPD) theory, but considering low affect and high affect students separately, we developed a system of five categories of video recommendations: 1) Watch new video; 2) Review current topic video with a new tutor; 3) Review segment of current video with current tutor; 4) Review segment of current video with a new tutor; 5) Watch next video in curriculum sequence. The category of recommendation was determined by student scores on a quiz and a sensor-free engagement detection model. New video recommendations (i.e., category 1) were selected based on a novel reinforcement learning algorithm that takes input from an item response theory model. The recommendation system was evaluated in a large field experiment, both before and after school closures due to the COVID-19 pandemic. The results show evidence of effectiveness of the video recommendation algorithm during the period of normal school operations, but the effect disappears after school closures. Implications for teacher orchestration of technology for normal classroom use and periods of school closure are discussed.","PeriodicalId":185465,"journal":{"name":"LAK22: 12th International Learning Analytics and Knowledge Conference","volume":"2011 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127371646","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Defining Productive Struggle in ST Math: Implications for Developing Indicators of Learning Behaviors and Strategies in Digital Learning Environments","authors":"Andrew E. Krumm, Andrew Coulson, J. Neisler","doi":"10.1145/3506860.3506901","DOIUrl":"https://doi.org/10.1145/3506860.3506901","url":null,"abstract":"This paper describes a process for operationally defining productive struggle in a widely used digital learning environment called ST Math. The process for developing an operational definition involved examining the existing literature for ways in which researchers have previously quantified productive struggle in digital learning environments. Using prior research, we defined productive struggle as a student persisting in a digital learning task while maintaining a likelihood of future success. To develop a machine-executable definition of productive struggle, we identified the typical number of attempts learners needed to complete a level in ST Math and applied a modified Performance Factors Analysis algorithm to estimate learners’ probability of success on a subsequent puzzle attempt within a level. Using definitions that differentially combined re-attempts and predicted probabilities, we examined the proportion of level attempts that could be newly classified as instances of productive struggle. The pragmatic approach described in this paper is intended to serve as an example for other digital learning environments seeking to develop indicators of productive struggle.","PeriodicalId":185465,"journal":{"name":"LAK22: 12th International Learning Analytics and Knowledge Conference","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127933087","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Needs Analysis and Prototype Evaluation of Student-facing LA Dashboard for Virtual Reality Content Creation","authors":"Tzi-Dong Jeremy Ng, Zuo Wang, Xiao Hu","doi":"10.1145/3506860.3506880","DOIUrl":"https://doi.org/10.1145/3506860.3506880","url":null,"abstract":"Being a promising constructionist pedagogy in recent years, maker education empowers students to take agency of their learning process through constructing both knowledge and real-world physical or digital products and fosters peer interactions for collective innovation. Learning Analytics (LA) excels at generating personalized, fine-grained feedback in near real-time and holds much potential in supporting process-oriented and peer-supported learning activities, including maker activities. In the context of virtual reality (VR) content creation for cultural heritage education, this study qualitatively solicited 27 students’ needs on progress monitoring, reflection, and feedback during their making process. Findings have inspired the prototype design of a student-facing LA dashboard (LAVR). Leveraging multimodal learning analytics (MmLA) such as text and audio analytics to fulfill students’ needs, the prototype has various features and functions including automatic task reminders, content quality detection, and real-time feedback on quality of audio-visual elements. A preliminary evaluation of the prototype with 10 students confirms its potential in supporting students’ self-regulated learning during the making process and for improving the quality of VR content. Implications on LA design for supporting maker education are discussed. Future work is planned to include implementation and evaluation of the dashboard in classrooms.","PeriodicalId":185465,"journal":{"name":"LAK22: 12th International Learning Analytics and Knowledge Conference","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128851216","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Barbara Šlibar, Jelena Gusic Mundjar, Sabina Rako, D. Šimić
{"title":"Co-occurrence patterns of issues and guidelines related to ethics and privacy of learning analytics in higher education—literature review","authors":"Barbara Šlibar, Jelena Gusic Mundjar, Sabina Rako, D. Šimić","doi":"10.1145/3506860.3506974","DOIUrl":"https://doi.org/10.1145/3506860.3506974","url":null,"abstract":"Ethics and privacy issues have been recognized as important factors for acceptance and trustworthy implementation of learning analytics. A large number of different issues has been recognized in the literature. Guidelines related to these issues are continuously being developed and discussed in research literature. The aim of this research was to identify patterns of co-occurrence of issues and guidelines in research papers discussing ethics and privacy issues, to gain better understanding of relationships between different ethics and privacy issues arising during implementation of learning analytics in higher education. A total of 93 papers published between 2010 and 2021 were qualitatively analyzed, and nine categories of issues and respective guidelines related to ethics and privacy in learning analytics were identified. Association rules mining Apriori algorithm was applied, where 93 papers represented transactions, and 18 categories of issues or guidelines (nine each) represented items. Two clusters of issues co-occurring in papers were identified, corresponding to deontology ethics (related to rules and duties), and consequentialism ethics (related to consequences of unethical behavior).","PeriodicalId":185465,"journal":{"name":"LAK22: 12th International Learning Analytics and Knowledge Conference","volume":"32 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117285668","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jiayu Li, Huiyong Li, Rwitajit Majumdar, Y. Yang, H. Ogata
{"title":"Self-directed Extensive Reading Supported with GOAL System: Mining Sequential Patterns of Learning Behavior and Predicting Academic Performance","authors":"Jiayu Li, Huiyong Li, Rwitajit Majumdar, Y. Yang, H. Ogata","doi":"10.1145/3506860.3506889","DOIUrl":"https://doi.org/10.1145/3506860.3506889","url":null,"abstract":"Self-directed learning (SDL) is an important skill in the 21st century, while the understanding of its process in behavior has not been well explored. Analysis of the sequential behavior patterns in SDL and the relations with students’ academic performance could help to advance our understanding of SDL in theory and practice. In this study, we mined the behavioral sequences of self-directed extensive reading from students’ learning and self-directed behavioral logs using differential pattern mining technique. Furthermore, we built models to predict students’ academic performance using the conventional behavior frequency features and the behavior sequence features. Experimental results identified 14 sequential patterns of SDL behaviors in the high-performance student group. The prediction model revealed the importance of sequential patterns in SDL behavior, which was built with an acceptable AUC. These findings suggested that several SDL strategies in behavior contribute to students’ academic performance, such as analysis learning status before planning, planning before learning, monitoring after learning.","PeriodicalId":185465,"journal":{"name":"LAK22: 12th International Learning Analytics and Knowledge Conference","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130978249","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Gloria Milena Fernández Nieto, Kirsty Kitto, S. B. Shum, Roberto Martínez Maldonado
{"title":"Beyond the Learning Analytics Dashboard: Alternative Ways to Communicate Student Data Insights Combining Visualisation, Narrative and Storytelling","authors":"Gloria Milena Fernández Nieto, Kirsty Kitto, S. B. Shum, Roberto Martínez Maldonado","doi":"10.1145/3506860.3506895","DOIUrl":"https://doi.org/10.1145/3506860.3506895","url":null,"abstract":"Learning Analytics (LA) dashboards have become a popular medium for communicating to teachers analytical insights obtained from student data. However, recent research indicates that LA dashboards can be complex to interpret, are often not grounded in educational theory, and frequently provide little or no guidance on how to interpret them. Despite these acknowledged problems, few suggestions have been made as to how we might improve the visual design of LA tools to support richer and alternative ways to communicate student data insights. In this paper, we explore three design alternatives to represent student multimodal data insights by combining data visualisation, narratives and storytelling principles. Based on foundations in data storytelling, three visual-narrative interfaces were designed with teachers: i) visual data slices, ii) a tabular visualisation, and iii) a written report. These were validated as a part of an authentic study where teachers explored activity logs and physiological data from co-located collaborative learning classes in the context of healthcare education. Results suggest that alternatives to LA dashboards can be considered as effective tools to support teachers’ reflection, and that LA designers should identify the representation type that best fits teachers’ needs.","PeriodicalId":185465,"journal":{"name":"LAK22: 12th International Learning Analytics and Knowledge Conference","volume":"21 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131763658","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}