{"title":"Experimental Design of Learning Analysis Dashboards for Teachers and Learners","authors":"Yassine Safsouf, K. Mansouri, F. Poirier","doi":"10.1145/3430895.3460990","DOIUrl":"https://doi.org/10.1145/3430895.3460990","url":null,"abstract":"Since learning in higher education is increasingly taking place online, the multiplication of web-based educational content, learning management systems (LMSs) and collaborative communication platforms have generated a large volume of data on learners and their learning activities. In recent years, interest has been growing in analyzing this data to support real-time decision making and improve the learning experience. This paper presents the results of a study conducted in higher education in Morocco, which evaluates a learning analysis dashboards (LADs) for both teachers and learners. The study shows that the dashboard, called TABAT, allowed a synthetic visualization of learning progress in courses and led to improved student engagement and success rates.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122873175","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Steven Ritter, N. Heffernan, J. Williams, Derek J. Lomas, K. Bicknell
{"title":"Second Workshop on Educational A/B Testing at Scale","authors":"Steven Ritter, N. Heffernan, J. Williams, Derek J. Lomas, K. Bicknell","doi":"10.1145/3430895.3460876","DOIUrl":"https://doi.org/10.1145/3430895.3460876","url":null,"abstract":"The emerging discipline of Learning Engineering is focused on putting into place tools and processes that use the science of learning as a basis for improving educational outcomes. An important part of Learning Engineering focuses on improving the effectiveness of educational software. In many software domains, A/B testing has become a prominent technique to achieve the software's goals. Many large companies (Amazon, Google, Facebook, etc.) run thousands of AB tests and present at the Annual Conference on Digital Experimentation (CODE), but that venue is too broad to address AB testing issues specific to EdTech platforms. We see a need to address issues with running large-scale A/B tests within the educational context, where the use of A/B testing lags other industries. This workshop will explore ways in which A/B testing in educational contexts differs from other domains and proposals to overcome current challenges so that this approach can become a more useful tool in the learning engineer's toolbox.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"2010 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114467859","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Sabrina Ludwig, A. Rausch, V. Deutscher, Jürgen Seifried
{"title":"Problem Solving Analytics (PSA) in the Web-Based Office Simulation LUCA","authors":"Sabrina Ludwig, A. Rausch, V. Deutscher, Jürgen Seifried","doi":"10.1145/3430895.3460877","DOIUrl":"https://doi.org/10.1145/3430895.3460877","url":null,"abstract":"Open-ended e-learning environments allow for explorative behaviour in challenging scenarios and hence, foster problem-solving competences. The web-based office simulation LUCA (funded by the German Federal Ministry of Education and Research) addresses the domain-specific competences of students in commercial vocational education and training (VET). The office simulation provides authentic office tools such as a spreadsheet application and an ERP software to solve complex work scenarios. These scenarios are implemented via the \"LUCA Editor\" and can contain automated assistance based on evidence rules of certain behaviours (\"scaffolding\"). The real-time analysis of the resulting log files enables the analysis of individual problem-solving behaviour (\"Problem Solving Analytics\", PSA). Teachers and trainers can monitor their students' problem-solving efforts in the \"LUCA Dashboard\", where they can also provide individual assistance via a chat tool. In our contribution, the scientific foundations of PSA will be outlined, followed by a demonstration of the latest prototype of LUCA and visitors' interaction with the software. LUCA's alpha version will be released in September of this year and will be available for practitioners in vocational schools and companies.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"24 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121594199","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Scale: A Window onto Universal Laws of Growth, Living and Dying in Organisms, Cities and Companies","authors":"G. West","doi":"10.1145/3430895.3462226","DOIUrl":"https://doi.org/10.1145/3430895.3462226","url":null,"abstract":"Despite its extraordinary complexity and diversity, many of Life's characteristics scale with size in a surprisingly simple fashion: time-scales from lifespans to growth-rates, and sizes from genome lengths to tree heights, scale systematically and predictably with size. Remarkably, cities and companies also exhibit systematic scaling: wages, profits, patents, crime, disease, and roads all scale in an approximately \"universal\" fashion across the globe. The origin of these laws, which constrain much of the organisation and dynamics of Life, will be explained and related to the underlying generic principles of the networks that sustain life ranging from circulatory systems of mammals to social networks of cities and companies. Their dynamics, which transcend history, geography and culture, have potentially dramatic implications for growth, development and global sustainability.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"24 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126559146","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Examining the Effects of Student Participation and Performance on the Quality of Learnersourcing Multiple-Choice Questions","authors":"Steven Moore, Huy A. Nguyen, John C. Stamper","doi":"10.1145/3430895.3460140","DOIUrl":"https://doi.org/10.1145/3430895.3460140","url":null,"abstract":"While generating multiple-choice questions has been shown to promote deep learning, students often fail to realize this benefit and do not willingly participate in this activity. Additionally, the quality of the student-generated questions may be influenced by both their level of engagement and familiarity with the learning materials. Towards better understanding how students can generate high quality questions, we designed and deployed a multiple-choice question generation activity in seven college-level online chemistry courses. From these courses, we collected data on student interactions and their contribution to the question-generation task. A total of 201 students enrolled in the courses and 57 of them elected to generate a multiple-choice question. Our results indicated that students were able to contribute quality questions, with 67% of them being evaluated by experts as acceptable for use. We further identified several student behaviors in the online courses that are correlated to their participation in the task and the quality of their contribution. Our findings can help teachers and students better understand the benefits of student-generated questions and effectively implement future learnersourcing activities.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"11 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116015166","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A Machine Learning Approach for Suggesting Feedback in Textual Exercises in Large Courses","authors":"Jan Philip Bernius, Stephan Krusche, B. Bruegge","doi":"10.1145/3430895.3460135","DOIUrl":"https://doi.org/10.1145/3430895.3460135","url":null,"abstract":"Open-ended textual exercises facilitate the comprehension of problem-solving skills. Students can learn from their mistakes when teachers provide individual feedback. However, courses with hundreds of students cause a heavy workload for teachers: providing individual feedback is mostly a manual, repetitive, and time-consuming activity. This paper presents CoFee, a machine learning approach designed to suggest computer-aided feedback in open-ended textual exercises. The approach uses topic modeling to split student answers into text segments and language embeddings to transform these segments. It then applies clustering to group the text segments by similarity so that the same feedback can be applied to all segments within the same cluster. We implemented this approach in a reference implementation called Athene and integrated it into Artemis. We used Athene to review 17 textual exercises in two large courses at the Technical University of Munich with 2,300 registered students and 53 teachers. On average, Athene suggested feedback for 26% of the submissions. Accordingly, 85% of these suggestions were accepted by the teachers, 5% were extended with a comment and then accepted, and 10% were changed.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"35 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126908891","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"With or Without EU: Navigating GDPR Constraints in Human Subjects Research in an Education Environment","authors":"Alex Duncan, David A. Joyner","doi":"10.1145/3430895.3460984","DOIUrl":"https://doi.org/10.1145/3430895.3460984","url":null,"abstract":"The General Data Protection Regulation (GDPR), passed in 2018, outlined new data privacy standards applying to residents of the European Union (EU). The impact of this law stretches beyond the EU to anyone - such as researchers across the world - collecting or processing data from EU residents. Researchers have had to augment their methodologies to ensure GDPR compliance. This initiative intersects with at-scale educational programs, which often enroll EU students and are also often the subject of institutional research. While creating a study to research students in an online Master of Science in Computer Science (MSCS) program, some of whom are in the EU, we encountered difficulties with ensuring GDPR compliance. This paper discusses the implications of the GDPR related to both general research and our specific study. We discuss the challenges of interpreting the GDPR and integrating it into our methodology as well as potential solutions, our ultimate resolution, and practical recommendations, and we consider what the future of data privacy legislation means for researchers.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"193 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115949165","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Karen D. Wang, S. Salehi, Max Arseneault, Krishnan Nair, C. Wieman
{"title":"Automating the Assessment of Problem-solving Practices Using Log Data and Data Mining Techniques","authors":"Karen D. Wang, S. Salehi, Max Arseneault, Krishnan Nair, C. Wieman","doi":"10.1145/3430895.3460127","DOIUrl":"https://doi.org/10.1145/3430895.3460127","url":null,"abstract":"Interactive simulations provide an exciting opportunity to assess and teach students the practices used by scientists and engineers to solve real-world problems. This study examines how the logged interaction data from a simulation-based task could be used to automate the assessment of complex problem-solving practices. A total of 73 college students worked on an interactive circuit puzzle embedded in a science simulation in an interview setting. Their problem-solving processes were videotaped and logged in the backend of the simulation. We extracted different sets of features from the log data and evaluated their effectiveness as predictors of students' problem-solving success and evidence for specific problem-solving practices. Our results indicate that the application of data mining techniques guided by knowledge gained from qualitative observation was instrumental in the discovery of semantically meaningful features from the raw log data. These knowledge-grounded features were significant predictors of students' overall problem-solving success and provided evidence on how well they adopted specific problem-solving practices, including decomposition, data collection, and data recording. The results point to promising directions for how scaffolding/feedback could be provided in educational simulations to enhance student learning in problem-solving skills.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127756809","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Comparative Analysis of the Feature Extraction Approaches for Predicting Learners Progress in Online Courses: MicroMasters Credential versus Traditional MOOCs","authors":"F. Soleimani, Jeonghyun Lee","doi":"10.1145/3430895.3460143","DOIUrl":"https://doi.org/10.1145/3430895.3460143","url":null,"abstract":"Although MicroMasters courses differ from traditional undergraduate level MOOCs in student demographics, course design, and outcomes, the various aspects of this type of program have not yet been sufficiently investigated. This study aims to pave the path towards enhancing the design of constituent courses of MicroMasters programs with the focus on the application of Machine Learning algorithms. Thereby, we use a large-scale clickstream edX database to explore the trends in the online engagement of learners in a MicroMasters program, detect clickstream events that are highly correlated with the students' progress, and investigate how the engagements differ from those in a classic individual MOOC. Contrary to the previous application of machine learning algorithms in learning analytics, we implement various well-known machine learning approaches such as stepwise regression and tree-based algorithms, evaluate their performance, and propose the best-performed approach. We elaborate on noticeable differences between the engagements of the considered two groups.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"60 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133550512","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
David Lang, Alex Wang, Nathan Dalal, A. Paepcke, M. Stevens
{"title":"Forecasting Undergraduate Majors Using Academic Transcript Data","authors":"David Lang, Alex Wang, Nathan Dalal, A. Paepcke, M. Stevens","doi":"10.1145/3430895.3460149","DOIUrl":"https://doi.org/10.1145/3430895.3460149","url":null,"abstract":"Committing to a major is a fateful step in an undergraduate's education, yet the relationship between courses taken early in an academic career and ultimate major choice remains little studied at scale. We analyze transcript data capturing the academic careers of 26,892 undergraduates at a private university between 2000 and 2020. We forecast students' terminal major on the basis of course-choice sequences beginning at university entry. We represent course enrollment history using natural-language methods and vector embeddings. We find that a student's very first enrolled course predicts their terminal major thirty times better than random guessing and more than a third better than majority class voting.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"37 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132277465","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}