Alejandra J. Magana;Syed Tanzim Mubarrat;Dominic Kao;Bedrich Benes
{"title":"基于人工智能的高等教育在线团队合作自动检测","authors":"Alejandra J. Magana;Syed Tanzim Mubarrat;Dominic Kao;Bedrich Benes","doi":"10.1109/TLT.2024.3456447","DOIUrl":null,"url":null,"abstract":"Fostering productive engagement within teams has been found to improve student learning outcomes. Consequently, characterizing productive and unproductive time during teamwork sessions is a critical preliminary step to increase engagement in teamwork meetings. However, research from the cognitive sciences has mainly focused on characterizing levels of productive engagement. Thus, the theoretical contribution of this study focuses on characterizing active and passive forms of engagement, as well as negative and positive forms of engagement. In tandem, researchers have used computer-based methods to supplement quantitative and qualitative analyses to investigate teamwork engagement. Yet, these studies have been limited to information extracted primarily from one data stream. For instance, text data from discussion forums or video data from recordings. We developed an artificial intelligence (AI)-based automatic system that detects productive and unproductive engagement during live teamwork sessions. The technical contribution of this study focuses on the use of three data streams from an interactive session: audio, video, and text. We automatically analyze them and determine each team's level of engagement, such as productive engagement, unproductive engagement, disengagement, and idle. The AI-based system was validated based on hand-coded data. We used the system to characterize productive and unproductive engagement patterns in teams using deep learning methods. Results showed that there were \n<inline-formula><tex-math>$>$</tex-math></inline-formula>\n91% prediction accuracy and \n<inline-formula><tex-math>$< $</tex-math></inline-formula>\n7% mismatches between predictions for the three engagement detectors. Moreover, Pearson's \n<inline-formula><tex-math>$r$</tex-math></inline-formula>\n values between the predictions of the three detectors were \n<inline-formula><tex-math>$>$</tex-math></inline-formula>\n0.844. On a scale of \n<inline-formula><tex-math>$-$</tex-math></inline-formula>\n1 (unproductive engagement) to 1 (productive engagement), the scores for all teams were 0.94 \n<inline-formula><tex-math>$\\pm$</tex-math></inline-formula>\n 0.04, suggesting high productive engagement. In addition, teams tended to mostly be in productive engagement before transitioning to disengagement (\n<inline-formula><tex-math>$>$</tex-math></inline-formula>\n90.34% of the time) and to idle (\n<inline-formula><tex-math>$>$</tex-math></inline-formula>\n93.69% of the time). Before transitioning to productive engagement, we noticed almost equal fractions of teams being in idle and disengagement modes. These results show that the system effectively detects engagement and can be a viable tool for characterizing productive and unproductive engagement patterns in teamwork sessions.","PeriodicalId":49191,"journal":{"name":"IEEE Transactions on Learning Technologies","volume":null,"pages":null},"PeriodicalIF":2.9000,"publicationDate":"2024-09-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10669806","citationCount":"0","resultStr":"{\"title\":\"AI-Based Automatic Detection of Online Teamwork Engagement in Higher Education\",\"authors\":\"Alejandra J. Magana;Syed Tanzim Mubarrat;Dominic Kao;Bedrich Benes\",\"doi\":\"10.1109/TLT.2024.3456447\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Fostering productive engagement within teams has been found to improve student learning outcomes. Consequently, characterizing productive and unproductive time during teamwork sessions is a critical preliminary step to increase engagement in teamwork meetings. However, research from the cognitive sciences has mainly focused on characterizing levels of productive engagement. Thus, the theoretical contribution of this study focuses on characterizing active and passive forms of engagement, as well as negative and positive forms of engagement. In tandem, researchers have used computer-based methods to supplement quantitative and qualitative analyses to investigate teamwork engagement. Yet, these studies have been limited to information extracted primarily from one data stream. For instance, text data from discussion forums or video data from recordings. We developed an artificial intelligence (AI)-based automatic system that detects productive and unproductive engagement during live teamwork sessions. The technical contribution of this study focuses on the use of three data streams from an interactive session: audio, video, and text. We automatically analyze them and determine each team's level of engagement, such as productive engagement, unproductive engagement, disengagement, and idle. The AI-based system was validated based on hand-coded data. We used the system to characterize productive and unproductive engagement patterns in teams using deep learning methods. Results showed that there were \\n<inline-formula><tex-math>$>$</tex-math></inline-formula>\\n91% prediction accuracy and \\n<inline-formula><tex-math>$< $</tex-math></inline-formula>\\n7% mismatches between predictions for the three engagement detectors. Moreover, Pearson's \\n<inline-formula><tex-math>$r$</tex-math></inline-formula>\\n values between the predictions of the three detectors were \\n<inline-formula><tex-math>$>$</tex-math></inline-formula>\\n0.844. On a scale of \\n<inline-formula><tex-math>$-$</tex-math></inline-formula>\\n1 (unproductive engagement) to 1 (productive engagement), the scores for all teams were 0.94 \\n<inline-formula><tex-math>$\\\\pm$</tex-math></inline-formula>\\n 0.04, suggesting high productive engagement. In addition, teams tended to mostly be in productive engagement before transitioning to disengagement (\\n<inline-formula><tex-math>$>$</tex-math></inline-formula>\\n90.34% of the time) and to idle (\\n<inline-formula><tex-math>$>$</tex-math></inline-formula>\\n93.69% of the time). Before transitioning to productive engagement, we noticed almost equal fractions of teams being in idle and disengagement modes. These results show that the system effectively detects engagement and can be a viable tool for characterizing productive and unproductive engagement patterns in teamwork sessions.\",\"PeriodicalId\":49191,\"journal\":{\"name\":\"IEEE Transactions on Learning Technologies\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":2.9000,\"publicationDate\":\"2024-09-09\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10669806\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"IEEE Transactions on Learning Technologies\",\"FirstCategoryId\":\"95\",\"ListUrlMain\":\"https://ieeexplore.ieee.org/document/10669806/\",\"RegionNum\":3,\"RegionCategory\":\"教育学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Transactions on Learning Technologies","FirstCategoryId":"95","ListUrlMain":"https://ieeexplore.ieee.org/document/10669806/","RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS","Score":null,"Total":0}
AI-Based Automatic Detection of Online Teamwork Engagement in Higher Education
Fostering productive engagement within teams has been found to improve student learning outcomes. Consequently, characterizing productive and unproductive time during teamwork sessions is a critical preliminary step to increase engagement in teamwork meetings. However, research from the cognitive sciences has mainly focused on characterizing levels of productive engagement. Thus, the theoretical contribution of this study focuses on characterizing active and passive forms of engagement, as well as negative and positive forms of engagement. In tandem, researchers have used computer-based methods to supplement quantitative and qualitative analyses to investigate teamwork engagement. Yet, these studies have been limited to information extracted primarily from one data stream. For instance, text data from discussion forums or video data from recordings. We developed an artificial intelligence (AI)-based automatic system that detects productive and unproductive engagement during live teamwork sessions. The technical contribution of this study focuses on the use of three data streams from an interactive session: audio, video, and text. We automatically analyze them and determine each team's level of engagement, such as productive engagement, unproductive engagement, disengagement, and idle. The AI-based system was validated based on hand-coded data. We used the system to characterize productive and unproductive engagement patterns in teams using deep learning methods. Results showed that there were
$>$
91% prediction accuracy and
$< $
7% mismatches between predictions for the three engagement detectors. Moreover, Pearson's
$r$
values between the predictions of the three detectors were
$>$
0.844. On a scale of
$-$
1 (unproductive engagement) to 1 (productive engagement), the scores for all teams were 0.94
$\pm$
0.04, suggesting high productive engagement. In addition, teams tended to mostly be in productive engagement before transitioning to disengagement (
$>$
90.34% of the time) and to idle (
$>$
93.69% of the time). Before transitioning to productive engagement, we noticed almost equal fractions of teams being in idle and disengagement modes. These results show that the system effectively detects engagement and can be a viable tool for characterizing productive and unproductive engagement patterns in teamwork sessions.
期刊介绍:
The IEEE Transactions on Learning Technologies covers all advances in learning technologies and their applications, including but not limited to the following topics: innovative online learning systems; intelligent tutors; educational games; simulation systems for education and training; collaborative learning tools; learning with mobile devices; wearable devices and interfaces for learning; personalized and adaptive learning systems; tools for formative and summative assessment; tools for learning analytics and educational data mining; ontologies for learning systems; standards and web services that support learning; authoring tools for learning materials; computer support for peer tutoring; learning via computer-mediated inquiry, field, and lab work; social learning techniques; social networks and infrastructures for learning and knowledge sharing; and creation and management of learning objects.