Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization最新文献

{"title":"Evaluating Pre-training Strategies for Collaborative Filtering","authors":"Júlio B. G. Costa, L. Marinho, Rodrygo L. T. Santos, Denis Parra","doi":"10.1145/3565472.3592949","DOIUrl":"https://doi.org/10.1145/3565472.3592949","url":null,"abstract":"Pre-training is essential for effective representation learning models, especially in natural language processing and computer vision-related tasks. The core idea is to learn representations, usually through unsupervised or self-supervised approaches on large and generic source datasets, and use those pre-trained representations (aka embeddings) as initial parameter values during training on the target dataset. Seminal works in this area show that pre-training can act as a regularization mechanism placing the model parameters in regions of the optimization landscape closer to better local minima than random parameter initialization. However, no systematic studies evaluate the effectiveness of pre-training strategies on model-based collaborative filtering. This paper conducts a broad set of experiments to evaluate different pre-training strategies for collaborative filtering using Matrix Factorization (MF) as the base model. We show that such models equipped with pre-training in a transfer learning setting can vastly improve the prediction quality compared to the standard random parameter initialization baseline, reaching state-of-the-art results in standard recommender systems benchmarks. We also present alternatives for the out-of-vocabulary item problem (i.e., items present in target but not in source datasets) and show that pre-training in the context of MF acts as a regularizer, explaining the improvement in model generalization.","PeriodicalId":309249,"journal":{"name":"Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization","volume":"66 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115438401","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"How Close are Predictive Models to Teachers in Detecting Learners at Risk?","authors":"Roberta Galici, Tanja Käser, G. Fenu, M. Marras","doi":"10.1145/3565472.3595620","DOIUrl":"https://doi.org/10.1145/3565472.3595620","url":null,"abstract":"Detecting learners in need of support is a complex process for both teachers and machines. Most prior work has devised visualization tools that allow teachers to do so by analyzing educational indicators. Other recent efforts have been devoted to models that predict whether learners might be at risk. However, the question on how teacher-like is the model behaving under this detection task still remains unanswered. In this paper, we investigate the (dis)agreement between teachers and model decisions, using a real-world flipped course as a case study. From the model perspective, we considered a well-known neural network, trained on educational indicators extracted from online pre-class logs. To gather teachers’ understanding, we employed a crowd sourcing approach including over 360 human intelligence tasks from 60 university teachers. We asked each recruited teacher to analyze visualizations pertaining to four relevant educational indicators of a given learner, and reason about their probability of failing the course (and so requiring support). Learners presented to teachers were selected to address different aspects of model confidence and (in)accuracy. Our results show that teacher and model predictions diverged for students who passed the course, while predictions were similar for students who failed the course. Moreover, confidence and correctness were more aligned in teachers than the model, reducing the unknown risks originally present in models. The source code is available at https://github.com/epfl-ml4ed/unknown-unknowns.","PeriodicalId":309249,"journal":{"name":"Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization","volume":"19 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115644781","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Combining Graph Neural Networks and Sentence Encoders for Knowledge-aware Recommendations","authors":"Giuseppe Spillo, C. Musto, Marco Polignano, P. Lops, Marco de Gemmis, G. Semeraro","doi":"10.1145/3565472.3592965","DOIUrl":"https://doi.org/10.1145/3565472.3592965","url":null,"abstract":"In this paper, we present a strategy to provide users with knowledge-aware recommendations based on the combination of graph neural networks and sentence encoders. In particular, our approach relies on the intuition that different data sources (i.e., structured data available in a knowledge graph and unstructured data, such as textual content) provide complementary information and can equally contribute to learn an accurate item representation. Accordingly, we first exploited graph neural networks to encode both collaborative features, such as the interactions between users and items, and structured properties of the items. Next, we used a sentence encoder that relies on transformers to learn a representation based on textual content describing the items. Finally, these embeddings are combined by exploiting a deep neural network where both self-attention and cross-attention mechanisms are used to learn the relationships between the initial embeddings and to further refine the representation. Such a neural network provides as output a prediction of users’ interest in the items, which is used to return a top-k recommendation list. In the experimental evaluation, we carried out an experiment against two datasets, and the results showed that our approach overcame several competitive baselines.","PeriodicalId":309249,"journal":{"name":"Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization","volume":"11 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125330438","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making","authors":"Sara Salimzadeh, Gaole He, U. Gadiraju","doi":"10.1145/3565472.3592959","DOIUrl":"https://doi.org/10.1145/3565472.3592959","url":null,"abstract":"Recent advances in the performance of machine learning algorithms have led to the adoption of AI models in decision making contexts across various domains such as healthcare, finance, and education. Different research communities have attempted to optimize and evaluate human-AI team performance through empirical studies by increasing transparency of AI systems, or providing explanations to aid human understanding of such systems. However, the variety in decision making tasks considered and their operationalization in prior empirical work, has led to an opacity around how findings from one task or domain carry forward to another. The lack of a standardized means of considering task attributes prevents straightforward comparisons across decision tasks, thereby limiting the generalizability of findings. We argue that the lens of ‘task complexity’ can be used to tackle this problem of under-specification and facilitate comparison across empirical research in this area. To retrospectively explore how different HCI communities have considered the influence of task complexity in designing experiments in the realm of human-AI decision making, we survey literature and provide an overview of empirical studies on this topic. We found a serious dearth in the consideration of task complexity across various studies in this realm of research. Inspired by Robert Wood’s seminal work on the construct, we operationalized task complexity with respect to three dimensions (component, coordinative, and dynamic) and quantified the complexity of decision tasks in existing work accordingly. We then summarized current trends and proposed research directions for the future. Our study highlights the need to account for task complexity as an important design choice. This is a first step to help the scientific community in drawing meaningful comparisons across empirical studies in human-AI decision making and to provide opportunities to generalize findings across diverse domains and experimental settings.","PeriodicalId":309249,"journal":{"name":"Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization","volume":"38 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125687392","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Comparative Analysis of Automatic Speech Recognition Errors in Small Group Classroom Discourse","authors":"Jie Cao, Ananya Ganesh, Jon Z. Cai, Rosy Southwell, E. M. Perkoff, Michael Regan, Katharina Kann, James H. Martin, Martha Palmer, S. D’Mello","doi":"10.1145/3565472.3595606","DOIUrl":"https://doi.org/10.1145/3565472.3595606","url":null,"abstract":"In collaborative learning environments, effective intelligent learning systems need to accurately analyze and understand the collaborative discourse between learners (i.e., group modeling) to provide adaptive support. We investigate how automatic speech recognition (ASR) errors influence discourse models of small group collaboration in noisy real-world classrooms. Our dataset consisted of 30 students recorded by consumer off-the-shelf microphones (Yeti Blue) while engaging in dyadic- and triadic- collaborative learning in a multi-day STEM curriculum unit. We found that two state-of-the-art ASR systems (Google Speech and OpenAI Whisper) yielded very high word error rates (0.822, 0.847) but very different profiles of error with Google being more conservative, rejecting 38% of utterances instead of 12% for Whisper. Next, we examined how these ASR errors influenced down-stream small group modeling based on pre-trained large language models for three tasks: Abstract Meaning Representation parsing (AMRParsing), on-task/off-task detection (OnTask), and Accountable Productive Talk prediction (TalkMove). As expected, models trained on clean human transcripts yielded degraded performance on all three tasks, measured by the transfer ratio (TR). However, the TR of the specific sentence-level AMRParsing task (.39 - .62) was much lower than that of the abstract discourse-level OnTask (.63- .94) and TalkMove tasks (.64-.72). Furthermore, different training strategies that incorporated ASR transcripts alone or as augmentations of human transcripts increased accuracy for the discourse-level tasks (OnTask and TalkMove) but not AMRParsing. Simulation experiments suggested that the models were tolerant of missing utterances in the dialog context, and that jointly improving ASR accuracy on important word classes (e.g., verbs and nouns) can improve performance across all tasks. Overall, our results provide insights into how different types of NLP-based tasks might be tolerant of ASR errors under extremely noisy conditions and provide suggestions for how to improve accuracy in small group modeling settings for a more equitable, engaging, and adaptive collaborative learning environment.","PeriodicalId":309249,"journal":{"name":"Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization","volume":"116 5 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128366584","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Bandit You Can Trust","authors":"Ethan Prihar, Adam C. Sales, N. Heffernan","doi":"10.1145/3565472.3592955","DOIUrl":"https://doi.org/10.1145/3565472.3592955","url":null,"abstract":"This work proposes Dynamic Linear Epsilon-Greedy, a novel contextual multi-armed bandit algorithm that can adaptively assign personalized content to users while enabling unbiased statistical analysis. Traditional A/B testing and reinforcement learning approaches have trade-offs between empirical investigation and maximal impact on users. Our algorithm seeks to balance these objectives, allowing platforms to personalize content effectively while still gathering valuable data. Dynamic Linear Epsilon-Greedy was evaluated via simulation and an empirical study in the ASSISTments online learning platform. In simulation, Dynamic Linear Epsilon-Greedy performed comparably to existing algorithms and in ASSISTments, slightly increased students’ learning compared to A/B testing. Data collected from its recommendations allowed for the identification of qualitative interactions, which showed high and low knowledge students benefited from different content. Dynamic Linear Epsilon-Greedy holds promise as a method to balance personalization with unbiased statistical analysis. All the data collected during the simulation and empirical study are publicly available at https://osf.io/zuwf7/.","PeriodicalId":309249,"journal":{"name":"Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization","volume":"130 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115233917","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Interactive Personalization of Classifiers for Explainability using Multi-Objective Bayesian Optimization","authors":"Suyog H. Chandramouli, Yifan Zhu, A. Oulasvirta","doi":"10.1145/3565472.3592956","DOIUrl":"https://doi.org/10.1145/3565472.3592956","url":null,"abstract":"Explainability is a crucial aspect of models which ensures their reliable use by both engineers and end-users. However, explainability depends on the user and the model’s usage context, making it an important dimension for user personalization. In this article, we explore the personalization of opaque-box image classifiers using an interactive hyperparameter tuning approach, in which the user iteratively rates the quality of explanations for a selected set of query images. Using a multi-objective Bayesian optimization (MOBO) algorithm, we optimize for both, the classifier’s accuracy and the perceived explainability ratings. In our user study, we found Pareto-optimal parameters for each participant, that could significantly improve explainability ratings of queried images while minimally impacting classifier accuracy. Furthermore, this improved explainability with tuned hyperparameters generalized to held-out validation images, with the extent of generalization being dependent on the variance within the queried images, and the similarity between the query and validation images. This MOBO-based method has the potential to be used in general to jointly optimize any machine learning objective along with any human-centric objective. The Pareto front produced after the interactive hyperparameter tuning can be useful during deployment, allowing for desired trade-offs between the objectives (if any) to be chosen by selecting the appropriate parameters. Additionally, user studies like ours can assess if commonly assumed trade-offs, such as accuracy versus explainability, exist in a given context.","PeriodicalId":309249,"journal":{"name":"Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization","volume":"98 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127234026","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Adaptive Context-Aware Planning Support for Students with Autism","authors":"Robin Cromjongh","doi":"10.1145/3565472.3595610","DOIUrl":"https://doi.org/10.1145/3565472.3595610","url":null,"abstract":"Managing university life is challenging for any student, including fast-paced self-reliant learning, first experiences with independent daily living, and the social demands of student life. These struggles are even more pronounced in students with Autism Spectrum Disorder (ASD), who might face additional difficulties, including interpersonal deficits, organisational challenges, lacking self-advocacy skills and sensory overload. This thesis uses a participatory design process to research how an adaptive, context-aware system could support students with autism in planning their student life, including managing time, tasks, stress and sensory stimulation. The first interviews resulted in a focus on intelligent planning systems with a low-effort interaction design. The design process will incrementally build towards intelligent support for personalised interactive scheduling as well as in-the-moment next-action recommendations that are context-aware and adaptive to data on the user’s stress level and sensory stimulation exposure.","PeriodicalId":309249,"journal":{"name":"Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization","volume":"14 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133766265","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Personalizing Time Loss Aversion to Reduce Social Media Use","authors":"Zi-yi Gao","doi":"10.1145/3565472.3592951","DOIUrl":"https://doi.org/10.1145/3565472.3592951","url":null,"abstract":"This study examines the effectiveness of a novel personalization approach for persuasive and behavior change systems: time loss aversion. Focusing on time instead of money, it influences behavior. Two interventions were developed and tested to reduce daily social media use and boost non-digital activities. Participants received information about their average daily social media use in terms of a week, month, and year, with one intervention offering substitute activities. Among 231 participants, both interventions successfully reduced intentions for future social media use, revealing generational effects, with Gen Zs using social media more than Millennials. These results have practical implications for personalizing interventions that reduce excessive digital engagement.","PeriodicalId":309249,"journal":{"name":"Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization","volume":"62 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114166966","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Human-centered Information Visualization Adaptation Engine","authors":"P. Andreou, Christos Amyrotos, Panagiotis Germanakos, I. Polycarpou","doi":"10.1145/3565472.3592961","DOIUrl":"https://doi.org/10.1145/3565472.3592961","url":null,"abstract":"Data Analytics is the art of turning data into insights for efficient and effective business decisions. Data visualization is among the most powerful tools in the data analyst’s arsenal, enabling the transformation of data into effective visualizations that can be easily comprehended. However, its effectiveness is often affected by the data analysts’ experience and their ability to quickly understand and interpret information. Even though business analytics tools have made a significant progress to deliver immersive data visualization environments for improving users’ efficiency and effectiveness, they still do not consider the individual differences in the core process that influences the visualization structure, encoding, and readability. This paper leverages the users’ individual differences to deliver a novel human-centered by-design adaptation engine for business users. The adaptation engine aims to improve the comprehension of data visualizations by delivering personalized content (visualization type and adaptation of visual elements), which in turn leads to improved accuracy and time-to-action efficiency. The proposed adaptation mechanism is evaluated using 45 professional business analysts from multiple industry sectors. The results suggest that individual differences can play an important role in the adaptation process of data visualizations enhancing analysts’ comprehensibility and decision making.","PeriodicalId":309249,"journal":{"name":"Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization","volume":"190 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114660726","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}