Mohi Reza, Juho Kim, Ananya Bhattacharjee, Anna N. Rafferty, J. Williams
{"title":"The MOOClet Framework: Unifying Experimentation, Dynamic Improvement, and Personalization in Online Courses","authors":"Mohi Reza, Juho Kim, Ananya Bhattacharjee, Anna N. Rafferty, J. Williams","doi":"10.1145/3430895.3460128","DOIUrl":"https://doi.org/10.1145/3430895.3460128","url":null,"abstract":"How can educational platforms be instrumented to accelerate the use of research to improve students' experiences? We show how modular components of any educational interface - e.g. explanations, homework problems, even emails - can be implemented using the novel MOOClet software architecture. Researchers and instructors can use these augmented MOOClet components for: (1) Iterative Cycles of Randomized Experiments that test alternative versions of course content; (2) Data-Driven Improvement using adaptive experiments that rapidly use data to give better versions of content to future students, on the order of days rather than months. A MOOClet supports both manual and automated improvement using reinforcement learning; (3) Personalization by delivering alternative versions as a function of data about a student's characteristics or subgroup, using both expert-authored rules and data mining algorithms. We provide an open-source web service for implementing MOOClets (www.mooclet.org) that has been used with thousands of students. The MOOClet framework provides an ecosystem that transforms online course components into collaborative micro-laboratories, where instructors, experimental researchers, and data mining/machine learning researchers can engage in perpetual cycles of experimentation, improvement, and personalization.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"20 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115561778","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Exploring Design Choices in Data-driven Hints for Python Programming Homework","authors":"T. Price, S. Marwan, J. Williams","doi":"10.1145/3430895.3460159","DOIUrl":"https://doi.org/10.1145/3430895.3460159","url":null,"abstract":"Students often struggle during programming homework and may need help getting started or localizing errors. One promising and scalable solution is to provide automated programming hints, generated from prior student data, which suggest how a student can edit their code to get closer to a solution, but little work has explored how to design these hints for large-scale, real-world classroom settings, or evaluated such designs. In this paper, we present CodeChecker, a system which generates hints automatically using student data, and incorporates them into an existing CS1 online homework environment, used by over 1000 students per semester. We present insights from survey and interview data, about student and instructor perceptions of the system. Our results highlight affordances and limitations of automated hints, and suggest how specific design choices may have impacted their effectiveness.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130352971","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Impact of Contextual Tips for Auto-Gradable Programming Exercises in MOOCs","authors":"Sebastian Serth, Ralf Teusner, C. Meinel","doi":"10.1145/3430895.3460166","DOIUrl":"https://doi.org/10.1145/3430895.3460166","url":null,"abstract":"Learners in Massive Open Online Courses offering practical programming exercises face additional challenges next to the actual course content. Beginners have to find approaches to deal with misconceptions and often struggle with the correct syntax while solving the exercises. The paper at hand presents insights from offering contextual tips in a web-based development environment used for practical programming exercises. We measured the effects of our approach in a Python course with 6,000 active students in a hidden A/B test and additionally used qualitative surveys. While a majority of learners valued the assistance, we were unable to show a direct impact on completion rates or average scores. We however noticed that users requesting tips took significantly longer and made more use of other assistance features of the platform than users in our control group. Insights from our study can be used to target beginners with more specific hints and provide additional, context-specific clues as part of the learning material.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"125 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129994988","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Akshay Dahiya, Rocko Graziano, India Irish, Thad Starner
{"title":"JackMarker with GitDown: A Framework to Counter Plagiarism at Scale","authors":"Akshay Dahiya, Rocko Graziano, India Irish, Thad Starner","doi":"10.1145/3430895.3460982","DOIUrl":"https://doi.org/10.1145/3430895.3460982","url":null,"abstract":"Plagiarism in computer science programs remains a problem at many universities. Programming assignments lend themselves to unpermitted collaboration: students can share code or find solutions from past semesters through simple internet searches. JackMarker helps address this problem by embedding a hidden, traceable, and unique token within documents. GitDown extends this work by locating available online solutions and requesting that they be removed.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"116 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129284000","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Zhi Liu, R. Mu, Shiqi Liu, Xian Peng, Sannyuya Liu
{"title":"Modeling Temporal Association of Cognition-Topic in MOOC Discussion to Track Learners' Cognitive Engagement Dynamics","authors":"Zhi Liu, R. Mu, Shiqi Liu, Xian Peng, Sannyuya Liu","doi":"10.1145/3430895.3460170","DOIUrl":"https://doi.org/10.1145/3430895.3460170","url":null,"abstract":"In the discussion forums of massive open online courses (MOOCs), cognitive processing (e.g., insight, certain) is considered an essential factor that can affect learners' learning outcomes, but the relationship between them has not been thoroughly investigated. Especially the dynamic nature of cognitive processing is still a significant research gap. In this study, we proposed an unsupervised topic model named Temporal Cognitive Topic Model (TCTM) to automatically classify cognitive processes and obtain the conditional probability with topics over time. The results indicated that completers had more active and timely cognitive engagement as time went on and tended to use certain cognitive words to discuss the topics related to the examination and certificates, which showed that they had explicit learning goals and plans. Non-completers often used exclusive cognitive words to discuss some off-task content that pointed out a distractive learning process. Using the model, teachers can capture learners' dynamic cognitive states and associated topics to improve teaching methods and increase course completion rates.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"65 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130493353","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"The Impact of Mobile Learning on Students' Self-Test Behavior in MOOCs","authors":"Max Bothe, C. Meinel","doi":"10.1145/3430895.3460151","DOIUrl":"https://doi.org/10.1145/3430895.3460151","url":null,"abstract":"Students can use personal mobile devices to access Massive Open Online Courses (MOOCs) in addition to desktop computers. However, user interfaces are often only scaled to smaller screen sizes and interaction patterns of a desktop learning experience do not always fit well with the characteristics of mobile devices. Adequate solutions for answering self-test questions on mobile devices often do not exist. In this paper, we explore the currently shown interaction patterns of MOOC learners when answering self-tests to make an informed decision about the requirements for the appropriate solution on mobile devices. The students' context was categorized into Desktop Web, Mobile Web, and Mobile Application. In an observational case study, the interaction events of two courses were analyzed regarding these device groups. Desktop Web is the most used environment. No practical differences between device groups were identified for subsequent attempts. Learners stick to a single device group and often only participate once in a self-test. Also, learners using mobile applications spend more time submitting self-tests.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"37 12","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"120965440","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Automatic Question Generation and the SmartStart Application","authors":"Bill Jerome, R. V. Campenhout, Benny G. Johnson","doi":"10.1145/3430895.3460878","DOIUrl":"https://doi.org/10.1145/3430895.3460878","url":null,"abstract":"Although content creators, instructors, and students alike see the value in providing interactive courseware to help students learn, they remain a costly solution to create. Platforms that allow the creation and delivery of interactive courseware are only fully utilized by those who can afford the labor cost of building hundreds of formative items just to start. Most of the time this means barriers are too high in either financial cost or extremely long development timelines. Our solution is called SmartStart, a process that works to reduce these barriers to creating courseware by automating basic steps that otherwise require significant manual labor. Automatic question generation (AQG) is one in a series of steps in the SmartStart process that work together to transform textbook content into a courseware learning environment.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"288 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121261207","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Setting up, Troubleshooting, and Innovating on the Delivery of Online Instruction: A Case Study of an LMS Q&A Forum","authors":"Kavana Ramesh, Laton Vermette, Parmit K. Chilana","doi":"10.1145/3430895.3460133","DOIUrl":"https://doi.org/10.1145/3430895.3460133","url":null,"abstract":"With the recent surge in online teaching, increasing numbers of instructors are using learning management systems (LMSs) to migrate their courses online and adapt their teaching methods. Given the steep learning curve for these feature-rich software applications, many instructors turn to online Q&A forums for help with learning and troubleshooting their LMS. We analyzed the content of 250 posts from a community-based Q&A forum for Canvas, a widely-used LMS, finding several recurring themes that illustrate how instructors are setting up courses for the first time, facilitating shared experiences between their students, and even seeking innovative ways to customize their course delivery. Our findings shed light on key barriers driving instructors to seek help and to what extent their help needs are actually addressed by the community. We discuss several opportunities for improving the design of LMSs and for better understanding and supporting instructors' custom needs to enhance their course delivery.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"213 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129433813","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Introducing Data Wrangling using Graphical Subgoals - Findings from an e-Learning Study","authors":"Lovisa Sundin, Q. Cutts","doi":"10.1145/3430895.3460155","DOIUrl":"https://doi.org/10.1145/3430895.3460155","url":null,"abstract":"Students across disciplines are increasingly wrangling their data programmatically. Previous research indicates that graphical, instructor-provided subgoals could help novices solve data wrangling tasks, and that graphical thumbnails could make the syntax documentation more usable. Using a 2x2 design, we trialled different versions of a web application for teaching data wrangling, which contains both programmatic and non-programmatic exercises. Half of all participants received graphical subgoals to visualise data wrangling solutions instead of just textual subgoals, and half received graphical thumbnails. Performance and time on task are reported, but did not result in any significant effect. We discuss possible reasons why.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"30 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129458064","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"What do I need to know? Designing Student Learning Tools that Aid Interaction with Recorded Lecture Content","authors":"Shipi Dhanorkar, D. Hellar, M. Rosson","doi":"10.1145/3430895.3460169","DOIUrl":"https://doi.org/10.1145/3430895.3460169","url":null,"abstract":"This paper presents an exploratory investigation of how students might benefit from recorded class lectures. Our overarching goal is to inform design of an artificial intelligence (AI) tool that adds value to lecture recordings. To study this possibility, we surveyed undergraduate students about their active learning strategies and their visions of how AI might help them to engage more richly with digitized learning materials. We report our findings and discuss implications for design of a tool that can add value to recorded lectures; we also consider more generally the possibilities of using lecture recordings to enhance student learning.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"51 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128630982","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}