Proceedings of the Eighth ACM Conference on Learning @ Scale最新文献

{"title":"Re-Design Classroom into MOOC-like Content With Remote Face-to-Face Sessions During the COVID-19 Pandemic: A Case Study in Graduate School","authors":"Shun Arima, Motoki Yasui, K. Okawa","doi":"10.1145/3430895.3460163","DOIUrl":"https://doi.org/10.1145/3430895.3460163","url":null,"abstract":"This paper introduces a case study that redesigns graduate school classrooms into massive open online course (MOOC)-like content with remote face-to-face sessions. Due to the influence of COVID-19, the higher-education environment is rapidly shifting to an online environment in many institutions. Therefore, instructors need to transform lessons they have been doing in physical classrooms to an online format. However, just conducting the same talks online is not enough to provide students with satisfactory learning. Instructors need to redesign class content and structure to optimize them for the online environment. In this case study, the research team and an instructor redesigned an existing graduate school class into MOOC-like content. We will discuss the specific methods and benefits of this initiative.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133238412","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Revisiting Transparency and Fairness in Algorithmic Systems Through the Lens of Public Education and Engagement","authors":"Motahhare Eslami","doi":"10.1145/3430895.3462228","DOIUrl":"https://doi.org/10.1145/3430895.3462228","url":null,"abstract":"The power, opacity, and bias of algorithmic systems have opened up new research areas for bringing transparency, fairness, and accountability into these systems. In this talk, I will revisit these lines of work, and argue that while they are critical to making algorithmic systems responsible, fresh perspectives are needed when these efforts fall short. I particularly discuss the necessity of algorithmic literacy and public education about the shortcomings of existing transparency and fairness efforts in algorithmic systems in order to enable everyday users to make more informed decisions in interactions with these systems. First, I discuss how algorithmic transparency, when not designed carefully, can be more harmful than helpful, and that we need to inform users about the limitations of transparency mechanisms provided in algorithmic systems. Second, I will talk about the current approaches tackling algorithmic bias in algorithmic systems, including bias detection and bias mitigation, and their limitations. I particularly show that the current algorithm auditing techniques that mainly rely on experts, and are conducted outside of everyday use of algorithmic systems, fall short in detecting biases that emerge in real-world contexts of use, and in the presence of complex social dynamics over time. This leads to the idea of \"everyday algorithm auditing\" that involves educating and enabling everyday users to understand, detect and/or interrogate biased and harmful algorithmic behaviors via their day-to-day interactions with algorithmic systems. I then take a new perspective on the bias mitigation efforts that endeavor to bring fairness to algorithmic systems, and argue that there are many cases that mitigating algorithmic bias is quite challenging, if not impossible. I propose the concept of \"bias transparency\" that centers bias awareness in algorithmic systems, particularly in high-stakes decision-making systems, by educating the public about potential biases these systems can introduce to users' decisions. I will end by discussing the importance of educating youth and fostering literacy around algorithmic systems from K-12 to prepare everyday users in their interactions with algorithmic systems.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"177 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123505751","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Components of Assessments and Grading At Scale","authors":"B. Eicher, David A. Joyner","doi":"10.1145/3430895.3460165","DOIUrl":"https://doi.org/10.1145/3430895.3460165","url":null,"abstract":"One of the major criticisms of efforts towards offering education at scale has been the Trap of Routine Assessment, the risk that student assessment will suffer from becoming excessively simplified in service of automation and scale. In this research, we examine the ways that students in an at-scale graduate program in computer science were assessed during their degrees. The program in question has scaled to over 10,000 students in only a few years, but awards a traditional Master's degree, providing the opportunity to investigate whether scale was achieved by transitioning to more routine assessment or by bringing scale to traditional strategies. To do this, we investigate the syllabi of 52 classes offered through the program to identify the types of assessments used, and we survey teaching teams for their approaches to evaluating these assessments. We merge this data with historical enrollment data to gain an overall summary of the kinds of assessments and evaluations received during their degrees. We ultimately find the program's scale has been managed by scaling up traditional assessment and evaluation strategies as the majority of grades are generated by human teaching teams based on projects and homeworks, with a relatively smaller portion generated exclusively by automated evaluation of exams.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"56 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125663030","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"What's In It for the Learners? Evidence from a Randomized Field Experiment on Learnersourcing Questions in a MOOC","authors":"Anjali Singh, Christopher A. Brooks, Yiwen Lin, Warren Li","doi":"10.1145/3430895.3460142","DOIUrl":"https://doi.org/10.1145/3430895.3460142","url":null,"abstract":"Question generation as a form of learnersourcing is both a metacognitive learning activity for students that encourages the development of higher-order thinking skills and a method for producing question banks and assessments. To better understand the motivations for learners who engage in learnersourcing and its impacts on student learning, we conducted an experiment that measured the effects of Multiple Choice Question (MCQ) generation in an introductory data science MOOC. We compared two approaches to question generation: (i) as a required activity, and (ii) as an optional activity. In both cases, the learnersourcing activity was part of the student summative evaluation. We found that learners value creating questions more, and create higher quality questions when they choose to do so compared to when it is required. At the same time there is a significant reduction in instructor evaluation workload in large-scale courses when learners engage by choice due to self-selection. Thus, we propose choice-based learnersourcing as a new form of scalable personalized learning design for MOOCs in particular. In addition, we contribute an exploration of the factors that influence learner choice to create (or not create) an MCQ, which can help contextualize the propensity of learners to engage in such learnersourcing activities.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"36 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128990880","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"What's the Point? How Scores Undermine Written Comments on Open-Ended Work","authors":"P. Crain, B. Bailey","doi":"10.1145/3430895.3460132","DOIUrl":"https://doi.org/10.1145/3430895.3460132","url":null,"abstract":"Scaling assessments typically relies on quantifying work quality, yet written comments are the assessment method of choice for open-ended work. Existing scalable solutions compromise by mapping quality scores to pre-authored comments, but how scores influence interpretation of these comments is not well understood. We report results from a study of how 441 participants authored and revised short stories in response to a score, written comments, both types of feedback, or no feedback. We analyzed data from the story-writing task and two surveys to determine task and feedback satisfaction, revision depth and effort, and improvement between drafts for each participant. We found task satisfaction and task performance were positively correlated among participants who were shown a score. Feedback satisfaction, revision effort, and improvement were highest among participants shown written comments. Either type of feedback prompted more deep revisions than no feedback, but together elicited fewer deep revisions than written comments alone. Our work informs the design of scalable open-ended assessment systems by contributing insights regarding how scores influence perceptions of written feedback and subsequent revision outcomes.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"45 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122865307","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Employing Peer Review to Evaluate the Quality of Student Generated Content at Scale: A Trust Propagation Approach","authors":"A. Darvishi, Hassan Khosravi, S. Sadiq","doi":"10.1145/3430895.3460129","DOIUrl":"https://doi.org/10.1145/3430895.3460129","url":null,"abstract":"Engaging students in the creation of learning resources has been demonstrated to have pedagogical benefits and lead to the creation of large repositories of learning resources which can be used to complement student learning in different ways. However, to effectively utilise a learnersourced repository of content, a selection process is needed to separate high-quality from low-quality resources as some of the resources created by students can be ineffective, inappropriate, or incorrect. A common and scalable approach to evaluating the quality of learnersourced content is to use a peer review process where students are asked to assess the quality of resources authored by their peers. However, this method poses the problem of \"truth inference\" since the judgements of students as experts-in-training cannot wholly be trusted. This paper presents a graph-based approach to propagate the reliability and trust using data from peer and instructor evaluations in order to simultaneously infer the quality of the learnersourced content and the reliability and trustworthiness of users in a live setting. We use empirical data from a learnersourcing system called RiPPLE to evaluate our approach. Results demonstrate that the proposed approach can propagate reliability and utilise the limited availability of instructors in spot-checking to improve the accuracy of the model compared to baseline models and the current model used in the system.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130617212","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Content-Neutral Immersive Environments for Cultivating Scalable Camaraderie","authors":"David A. Joyner, Akhil Mavilakandy, Ishaani Mittal, Denise G. Kutnick, B. MacIntyre","doi":"10.1145/3430895.3460136","DOIUrl":"https://doi.org/10.1145/3430895.3460136","url":null,"abstract":"Drawbacks of online education, especially at scale, include the isolation and loss of community that come with taking classes from one's home rather than in a typical coordinated classroom. In this series of studies, we explore the potential of an immersive virtual environment to recreate shared educational contexts in an online program, emphasizing those that focus on universal dynamics like discussions, Q&A, and lecture-watching. Using Mozilla Hubs-which offers virtual reality as well as a browser-based immersive environment for participants without VR headsets-we implement three common, student-driven on-campus environments: a lecture hall, a student lounge, and a poster session. We test these environments with students in an online graduate program through surveys and a controlled experiment. We find significant interest in the potential of these environments, but hurdles to their adoption as well.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"30 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123376315","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Peer-Grading at Scale with Rank Aggregation","authors":"Lin Ling, Chee-Wei Tan","doi":"10.1145/3430895.3460980","DOIUrl":"https://doi.org/10.1145/3430895.3460980","url":null,"abstract":"Thanks to the wide availability of the internet and personal computing devices, online teaching methods like Massive Open Online Courses (MOOC) are becoming an essential part of modern education. While these methods enable educators to reach much more students, the massive volume of assignments to grade places a heavy burden on the instructors. Most online courses remedy this by restricting the question types to simple forms or performing naive peer-grading. These approaches are either too restricted to capture students' learning level, or require heavy supervision from the instructors to ensure the grades are fair. In this paper, we propose a rank-aggregation-based peer-grading method that estimates the quality of each assignment and the probability that each student is grading unbiasedly. The estimation errors have theoretical upper-bounds, and the bounds can be proved to tighten when the problem size increases, which is confirmed by our numerical experiment.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129592384","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"How Online Learners Build Cognitive Presence: Implications from a Machine Learning Approach","authors":"J. Hosmer, Jeonghyun Lee","doi":"10.1145/3430895.3460986","DOIUrl":"https://doi.org/10.1145/3430895.3460986","url":null,"abstract":"Within the last year, the number of students who participate in online learning platforms has exploded. The transition to online platforms has moved discussion out of the classroom and onto digital spaces. To support this transition we aim to create a model which can help teachers and teaching assistants understand how the discourse of learners in discussion forums evolve through multiple phases of cognitive presence over time.To this end, we use cutting-edge natural language processing techniques and apply machine-learning algorithms to build a model that can predict the cognitive presence phase of posts based on the community of inquiry framework.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124459153","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Teaching the Masses on Twitch: An Initial Exploration of Educational Live-Streaming","authors":"H. Steinbeck, Ralf Teusner, C. Meinel","doi":"10.1145/3430895.3460157","DOIUrl":"https://doi.org/10.1145/3430895.3460157","url":null,"abstract":"Streaming games and entertainment content are established formats on large portals like YouTube and Twitch. Educational streams have not yet reached the same popularity. Consequently, the existing research of gaming streams by far exceeds the research of live-streaming lectures. In this paper, we contribute first insights regarding this area, outline the status quo of the edu-streaming ecosystem, and highlight common approaches and characteristics. Through a descriptive study of 100 popular gaming streams, we systemize the inductively found features and approaches that are seen in both ecosystems. With a further focus on 20 educational streams, we highlight features such as synchronous community building and on-stream interactivity. We project the main differences between the core characteristics of MOOCs and educational streams. Finally, we propose further research directions for the emerging field of public, synchronous online education.","PeriodicalId":125581,"journal":{"name":"Proceedings of the Eighth ACM Conference on Learning @ Scale","volume":"4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122733347","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}