Sreecharan Sankaranarayanan, Siddharth Reddy Kandimalla, C. Bogart, R. C. Murray, Michael Hilton, M. Sakr, C. Rosé
{"title":"Combining Collaborative Reflection based on Worked-Out Examples with Problem-Solving Practice: Designing Collaborative Programming Projects for Learning at Scale","authors":"Sreecharan Sankaranarayanan, Siddharth Reddy Kandimalla, C. Bogart, R. C. Murray, Michael Hilton, M. Sakr, C. Rosé","doi":"10.1145/3430895.3460152","DOIUrl":"https://doi.org/10.1145/3430895.3460152","url":null,"abstract":"Computer science pedagogy has overwhelmingly favored problem-solving practice over methods of engagement like worked-out example study especially in advanced classes. This is due to the belief that while these alternative methods may improve student conceptual learning, they may leave them less able to perform on authentic problem-solving tasks from a lack of hands-on practice. In this paper, we perform a direct comparison of this trade-off in a synchronous collaborative programming project by adjusting the boundary between problem-solving and collaborative reflection based on a worked-out example while keeping the total time on task constant. We find that the more time students spent on worked example study, the more was the observed improvement in the pre- to post-test scores with no significant difference in performance on a subsequent problem-solving task. These results, therefore, challenge the dominant place of problem-solving practice in the advanced curricular context and inform the design of collaborative programming projects at scale.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"26 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116748640","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jinjin Zhao, Candace Thille, Neelesh Gattani, D. Zimmaro
{"title":"A Novel Framework for Discovering Cognitive Models of Learning","authors":"Jinjin Zhao, Candace Thille, Neelesh Gattani, D. Zimmaro","doi":"10.1145/3430895.3460156","DOIUrl":"https://doi.org/10.1145/3430895.3460156","url":null,"abstract":"A cognitive model is a descriptive account or computational representation of human thinking about a given concept, skill, or domain. A cognitive model of learning, includes both a way of organizing knowledge within a subject area and an account of how humans develop accurate and complete knowledge of that subject area. Learning designers engage in a variety of practices to unpack knowledge from subject matter experts and novices to develop cognitive models of learning and use those models to guide the design of instruction or instructional technologies. Traditional approaches to eliciting and organizing knowledge, such as conducting a cognitive task analysis (CTA) [10] with experts and novices, are labor-intensive and require specific expertise that many learning designers do not have. However, learning data generated from learners' interaction with the courses, reveal how humans think about and develop knowledge. We propose a novel framework that uses learning data to discover and refine cognitive models of learning. The framework includes a Variational Autoencoder (VAE) module and a Gaussian Mixture Model (GMM) module. We provide one case study in a corporate setting to demonstrate the effectiveness of the proposed framework compared to other approaches.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"32 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123859417","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Sophia Yang, Ziyuan Wei, Geoffrey L. Herman, Abdussalam Alawini
{"title":"Analyzing Patterns in Student SQL Solutions via Levenshtein Edit Distance","authors":"Sophia Yang, Ziyuan Wei, Geoffrey L. Herman, Abdussalam Alawini","doi":"10.1145/3430895.3460979","DOIUrl":"https://doi.org/10.1145/3430895.3460979","url":null,"abstract":"Structured Query Language (SQL), the standard language for relational database management systems, is an essential skill for software developers, data scientists, and professionals who need to interact with databases. SQL is highly structured and presents diverse ways for learners to acquire this skill. However, despite the significance of SQL to other related fields, little research has been done to understand how students learn SQL as they work on homework assignments. In this paper, we analyze students' SQL submissions to homework problems of the Database Systems course offered at the University of Illinois at Urbana-Champaign. For each student, we compute the Levenshtein Edit Distances between every submission and their final submission to understand how students reached their final solution and how they overcame any obstacles in their learning process. Our system visualizes the edit distances between students' submissions to a SQL problem, enabling instructors to identify interesting learning patterns and approaches. These findings will help instructors target their instruction in difficult SQL areas for the future and help students learn SQL more effectively.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"128 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124543793","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Co-Evolution of Human Capabilities and Intelligent Technologies for Digital Education","authors":"Sanna Järvelä","doi":"10.1145/3430895.3462227","DOIUrl":"https://doi.org/10.1145/3430895.3462227","url":null,"abstract":"There is much interest to advance digital technologies supporting teaching, learning and education. Yet, many ideas, e.g., implementing data and artificial intelligence in education, still lack systematic understanding of human learning process. Also, new kind of capabilities are needed that are necessary to succeed in a rapidly changing world. In my talk I introduce recent advancements in research on socially shared regulation in learning which provides a framework for developing these competences. I discuss the role of technology in understanding and supporting socially shared regulation and conclude with future perspective how co-evolution of human capabilities and technologies can be enhanced for digital education.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129148125","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
J. Cho, Yue Li, Anne K. Armstrong, A. Russ, M. Krasny, René F. Kizilcec
{"title":"Using Social Norms to Promote Actions Beyond the Course","authors":"J. Cho, Yue Li, Anne K. Armstrong, A. Russ, M. Krasny, René F. Kizilcec","doi":"10.1145/3430895.3460144","DOIUrl":"https://doi.org/10.1145/3430895.3460144","url":null,"abstract":"Educators and researchers in online education have grappled with not only how to increase course completion but also how to make a broader impact that goes beyond online courses, such as course participants' real-world applications of the learned knowledge and skills. Research in social psychology and behavioral science suggests that social norms interventions, which convey norms shared in the community that people belong in to promote desirable behaviors, can offer a low-cost and scalable approach to encourage actions beyond the courses (ABCs). We tested three social norm interventions that presented a weekly normative message (descriptive, dynamic, or injunctive norm) with aggregate information about course participants' ABCs in the prior week. Randomized experiments in three online courses found effects on ABCs to be weak and moderated by norm message type and the complexity of the target behavior. Although the interventions did not improve course completion, the dynamic norm message was more effective at promoting ABCs for complex behaviors, such as developing environmental education activities.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"48 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124777643","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Toward Reshaping the Syllabus for Education at Scale","authors":"B. Eicher, David A. Joyner","doi":"10.1145/3430895.3460987","DOIUrl":"https://doi.org/10.1145/3430895.3460987","url":null,"abstract":"Ensuring that students are fully informed about course content and policies is always a challenge, but online education at scale adds additional complications. In this paper we present observations about the place of the syllabus in education at scale, based on the actual syllabus documents from 48 courses in a Computer Science Master's degree program offered online and at scale. On the basis of these observations, we offer preliminary recommendations for factors that instructors should keep in mind when they compile a syllabus for similar courses.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"73 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131815023","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Benjamin Xie, Matthew J. Davidson, Baker Franke, Emily M. McLeod, Min Li, Amy J. Ko
{"title":"Domain Experts' Interpretations of Assessment Bias in a Scaled, Online Computer Science Curriculum","authors":"Benjamin Xie, Matthew J. Davidson, Baker Franke, Emily M. McLeod, Min Li, Amy J. Ko","doi":"10.1145/3430895.3460141","DOIUrl":"https://doi.org/10.1145/3430895.3460141","url":null,"abstract":"Understanding inequity at scale is necessary for designing equitable online learning experiences, but also difficult. Statistical techniques like differential item functioning (DIF) can help identify whether items/questions in an assessment exhibit potential bias by disadvantaging certain groups (e.g. whether item disadvantages woman vs man of equivalent knowledge). While testing companies typically use DIF to identify items to remove, we explored how domain-experts such as curriculum designers could use DIF to better understand how to design instructional materials to better serve students from diverse groups. Using Code.org's online Computer Science Discoveries (CSD) curriculum, we analyzed 139,097 responses from 19,617 students to identify DIF by gender and race in assessment items (e.g. multiple choice questions). Of the 17 items, we identified six that disadvantaged students who reported as female when compared to students who reported as non-binary or male. We also identified that most (13) items disadvantaged AHNP (African/Black, Hispanic/Latinx, Native American/Alaskan Native, Pacific Islander) students compared to WA (white, Asian) students. We then conducted a workshop and interviews with seven curriculum designers and found that they interpreted item bias relative to an intersection of item features and student identity, the broader curriculum, and differing uses for assessments. We interpreted these findings in the broader context of using data on assessment bias to inform domain-experts' efforts to design more equitable learning experiences.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130460972","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Generating Response-Specific Elaborated Feedback Using Long-Form Neural Question Answering","authors":"A. Olney","doi":"10.1145/3430895.3460131","DOIUrl":"https://doi.org/10.1145/3430895.3460131","url":null,"abstract":"In contrast to simple feedback, which provides students with the correct answer, elaborated feedback provides an explanation of the correct answer with respect to the student's error. Elaborated feedback is thus a challenge for AI in education systems because it requires dynamic explanations, which traditionally require logical reasoning and knowledge engineering to generate. This study presents an alternative approach that formulates elaborated feedback in terms of long-form question answering (LFQA). An off-the-shelf LFQA system was evaluated by human raters in a 2x2x2x2 ablation design that manipulated the context documents given to the LFQA model and the post-processing of model output. Results indicate that context manipulations improve performance but that post-processing can have detrimental results.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"146 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123377645","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Personalized Lecture Recommendations to Facilitate Bite-Sized Learning","authors":"Meltem Tutar, Austin Wang, Gulsen Kutluoglu","doi":"10.1145/3430895.3460988","DOIUrl":"https://doi.org/10.1145/3430895.3460988","url":null,"abstract":"This paper presents an unsupervised, content-based approach to match users with shorter pieces of specific learning content, lectures, to target their learning goals at a more granular level. This method is especially useful when implicit data is unreliable or limited. At a high level, our approach generates a set of lectures for every topic via clustering and then matches lectures to users via users' topic affinities. Our central hypothesis is that important, fundamental concepts are repeated within many courses on the same topic, and by extracting clusters, we can identify these key information lectures per topic.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"19 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114555294","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Assessing Function Names and Quantifying the Relationship Between Identifiers and Their Functionality to Improve Them","authors":"Charis Charitsis, C. Piech, John C. Mitchell","doi":"10.1145/3430895.3460161","DOIUrl":"https://doi.org/10.1145/3430895.3460161","url":null,"abstract":"When students first learn to program, they often focus on functionality: does a program work? In an era where software volume and complexity increase exponentially, it is equally important that they learn to write code with style. Quality code starts with the building blocks for any program, its functions. A carefully chosen name is vital for program maintainability and manageability. The identifier is the most portable and concise way to summarize what the function does. What makes for the right choice? And can we automatically assess the quality of function names? Using natural language processing, we were able to create a probabilistic model to evaluate their clarity. Using functionality encodings, we attempt to learn the relationship between functions in different programs to improve their names. We analyzed a total of 3,900 programs tackling three novice programming tasks submitted by 1,300 students in CS1.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"119 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116317605","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}