Companion Publication of the 2021 International Conference on Multimodal Interaction最新文献

{"title":"Predicting Worker Accuracy from Nonverbal Behaviour: Benefits and Potential for Algorithmic Bias","authors":"Yuushi Toyoda, Gale M. Lucas, J. Gratch","doi":"10.1145/3461615.3485427","DOIUrl":"https://doi.org/10.1145/3461615.3485427","url":null,"abstract":"With the rise of algorithmic management (and online work in general), there is a growing interest in techniques that can monitor worker performance. For example, if a system can automatically detect whether a worker is becoming distracted or disengaged, it can intervene to motivate the worker or flag their output as requiring further quality control. Prior research has explored the potential for detecting nonverbal cues that could predict mistakes (e.g., detecting boredom in students or fatigue in drivers). Here, we learn a model that reliably predicts worker accuracy from nonverbal behaviour during tedious and repetitive annotation tasks. We show that the annotation accuracy can be substantially improved by discarding annotations that are predicted to be of low accuracy by the model. While these results are promising, recent concerns about algorithmic bias led us to further investigate whether the accuracy is influenced by skin tone. Unfortunately, we find that the algorithm showed systematic bias that disadvantaged some dark-skinned workers and incorrectly rewarded some with lighter-skin. We discuss the apparent reasons for this bias and suggestions for how and if such methods could be deployed to enhance worker engagement.","PeriodicalId":440830,"journal":{"name":"Companion Publication of the 2021 International Conference on Multimodal Interaction","volume":"4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125035708","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Automatic analysis of infant engagement during play: An end-to-end learning and Explainable AI pilot experiment","authors":"Marc Fraile, Joakim Lindblad, Christine Fawcett, Natasa Sladoje, Ginevra Castellano","doi":"10.1145/3461615.3485443","DOIUrl":"https://doi.org/10.1145/3461615.3485443","url":null,"abstract":"Infant engagement during play is an active area of research, related to the development of cognition. Automatic detection of engagement could benefit the research process, but existing techniques used for automatic affect detection are unsuitable for this scenario, since they rely on the automatic extraction of facial and postural features trained on clear video capture of adults. This study shows that end-to-end Deep Learning methods can successfully detect engagement of infants, without the need of clear facial video, when trained for a specific interaction task. It further shows that attention mapping techniques can provide explainability, thereby enabling trust and insight into a model’s reasoning process.","PeriodicalId":440830,"journal":{"name":"Companion Publication of the 2021 International Conference on Multimodal Interaction","volume":"49 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125900620","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Web-Based Longitudinal Mental Health Monitoring System","authors":"Zhiwei Chen, Weizhao Yang, Jinrong Li, Jiale Wang, Shuai Li, Ziwen Wang, Lei Xie","doi":"10.1145/3461615.3491113","DOIUrl":"https://doi.org/10.1145/3461615.3491113","url":null,"abstract":"Current clinical assessments of depression disorder are heavily relied on the questionnaire tables on patients’ daily behavior, sleeping, and mood status of the past two weeks. However, the information obtained through the patient’s review of the past two weeks’ experience is neither timely nor objective. Moreover, while patients have medicine at home, doctors lose the way of monitoring and intervening them on time. In this paper, we propose and implement a web-based longitudinal mental health monitoring system. On the user end, the patients can report their daily information through ecological momentary assessment (EMA), share their emotions in speech or face video, test their depression severity through the PHQ-9 questionnaire table or face videos recorded while going through a semi-structured interview, and check their recent history of activity, sleeping, emotion log, and depression severity etc. The server end implements emotion recognition and depression estimation on the pre-trained deep learning models. On the doctor end, the doctor can manage the information of all the patients under his(her) supervision, monitor their recent status, and edit their depression severity after clinical diagnosis.","PeriodicalId":440830,"journal":{"name":"Companion Publication of the 2021 International Conference on Multimodal Interaction","volume":"45 1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122683496","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Uncertainties based queries for Interactive policy learning with evaluations and corrections","authors":"C. Celemin, J. Kober","doi":"10.1145/3461615.3485404","DOIUrl":"https://doi.org/10.1145/3461615.3485404","url":null,"abstract":"","PeriodicalId":440830,"journal":{"name":"Companion Publication of the 2021 International Conference on Multimodal Interaction","volume":"21 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117261695","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Get Together in the Middle-earth: a First Step Towards Hybrid Intelligence Systems","authors":"G. Varni, André-Marie Pez, M. Mancini","doi":"10.1145/3461615.3485413","DOIUrl":"https://doi.org/10.1145/3461615.3485413","url":null,"abstract":"In the last decade, the number of computer systems using AI has increased dramatically. To date, indeed, AI is present in almost all the aspects of the human everyday life. This resulted in the attempt of scholars in Computer Science to endow machines with human-like socio-cognitive skills and/or human-like embodiment to try to improve interactions. Such an approach, however, highlights several crucial issues related to the substantial differences between fine-grained human skills and what machines can do and learn. So, although being expensive and sophisticated tools, machines tend to be “idiots savants”. Hybrid Intelligence (HI) is aimed to tackle this issue by proposing, as Akata and colleagues say, “systems that operate as mixed teams, where humans and machines cooperate synergistically, proactively, and purposefully to achieve shared goals”. To our knowledge, however, HI is at a very early exploratory stage, and few concrete solutions to deal with it exist. In this position paper we introduce and briefly describe “Middle-Earth”, a conceptual and experimental ground to study HI. Moreover, we present a first prototype of a software platform based on immersive VR environments, on which we plan to carry out in the future the first pioneering experiments on teams of humans and/or AI-driven agents getting together in Middle Earth to perform collaborative tasks.","PeriodicalId":440830,"journal":{"name":"Companion Publication of the 2021 International Conference on Multimodal Interaction","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129957916","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Facial Micro-Expression Recognition Based on Multi-Scale Temporal and Spatial Features","authors":"Hao Zhang, B. Liu, J. Tao, Zhao Lv","doi":"10.1145/3461615.3491107","DOIUrl":"https://doi.org/10.1145/3461615.3491107","url":null,"abstract":"Micro-expression is a kind of facial activity with weak changes and short duration that can reflect people’s true feelings. For micro-expression recognition, it is not only necessary to extract the spatial feature information of the face movement changes in the image, but also to consider the time series information of the continuous image sequence. Thus, we propose a multiple aggregation networks to verify the impact of local facial regions and temporal features on micro-expression recognition in detail. It can learn the temporal and spatial feature from the whole micro expression video frame and combined the local region where the micro-expression mainly occurs with the global region. The spatial features of micro-expressions frames are extracted by 3D CNN, and the extracted video sequences features are input into LSTM processing temporal features. Experiments from two public datasets, CASME-II and SAMM, show that our method obtains higher performance than several existing studies.","PeriodicalId":440830,"journal":{"name":"Companion Publication of the 2021 International Conference on Multimodal Interaction","volume":"53 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126676098","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Temporal Attentive Adversarial Domain Adaption for Cross Cultural Affect Recognition","authors":"Haifeng Chen, Yifan Deng, D. Jiang","doi":"10.1145/3461615.3491110","DOIUrl":"https://doi.org/10.1145/3461615.3491110","url":null,"abstract":"Continuous affect recognition is becoming an increasingly attractive research topic, recent works mainly focus on modeling the temporal dependency and multi-modal fusion to boost the performance. Despite recent improvement, the cross-cultural affect recognition in videos is still not well-explored. In this paper, we propose the temporal attentive adversarial domain adaption for cross cultural affect recognition. The LSTM is firstly used to encode the contextual representation for each frame. Then, a DNN based regressor is used to estimate the affective dimension arousal or valence, and optimized to promote the encoded representation is emotion discriminative. In addition, another DNN based sequence level culture classifier, which takes the fused representation of each frame as the input, is used to recognize the culture of the input sequence, and optimized to encourage the encoded representation is culture invariant. Since different frames over a video may contribute not equally in recognizing the culture, we propose to add another frame level culture classifier, which could adaptively and attentively assign more weighting scores for the important frames for recognizing the culture. The proposed method is evaluated on the benchmark dataset AVEC2019 CES. Our experimental results show that the proposed method improves the performance compared to state-of-the-art methods, with the concordance correlation coefficient (CCC) reaching 0.576 for arousal and 0.472 for valence, on the cross cultural test set.","PeriodicalId":440830,"journal":{"name":"Companion Publication of the 2021 International Conference on Multimodal Interaction","volume":"57 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126233202","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Multimodal Dynamic Neural Network for Call for Help Recognition in Elevators","authors":"Ran Ju, Huangrui Chu, Yechen Wang, Qi Deng, Ming Cheng, Ming Li","doi":"10.1145/3461615.3491112","DOIUrl":"https://doi.org/10.1145/3461615.3491112","url":null,"abstract":"As elevator accidents do great damage to people’s lives and property, taking immediate responses to emergent calls for help is necessary. In most emergency cases, passengers must use the “SOS” button to contact the remote safety guard. However, this method is unreliable when passengers lose the ability of body movement. To address this problem, we define a novel task of identifying real and fake calls for help in elevator scenes. Given that the limited call for help dataset collected in elevators contains multimodal data of real and fake categories, we collected and constructed an audiovisual dataset dedicated to the proposed task. Moreover, we present a novel instance-modality-wise dynamic framework to efficiently use the information from each modality and make inferences. Experimental results show that our multimodal network improves the performance on the call for help multimodal dataset by 2.66% (accuracy) and 1.25% (F1 Score) with respect to the pure audio model. Besides, our method outperforms other methods on our dataset.","PeriodicalId":440830,"journal":{"name":"Companion Publication of the 2021 International Conference on Multimodal Interaction","volume":"62 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126405462","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Self-assessed Emotion Classification from Acoustic and Physiological Features within Small-group Conversation","authors":"Woan-Shiuan Chien, Huang-Cheng Chou, Chi-Chun Lee","doi":"10.1145/3461615.3485411","DOIUrl":"https://doi.org/10.1145/3461615.3485411","url":null,"abstract":"Individual (personalized) self-assessed emotion recognition has recently received more attention, such as Human-Centered Artificial Intelligence (AI). In most previous studies, researchers utilized the physiological changes and reactions in the body evoked by multi-media stimuli, e.g., video or music, to build a model for recognizing individuals’ emotions. However, this elicitation approach is less impractical in the human-human interaction because the conversation is dynamic. In this paper, we firstly investigate the individual emotion recognition task under three-person small group conversations. While predicting personalized emotions from physiological signals is well-studied, few studies focus on emotion classification (e.g., happiness and sadness). Most prior works only focus on binary dimensional emotion recognition or regression, such as valence and arousal. Hence, we formulate the individual emotion recognition task into an individual-level emotion classification. In the proposed method, we consider the physiological changes in each individual’s body and acoustic turn-taking dynamics during group conversations for predicting individual emotions. Meanwhile, we assume that the emotional states of humans might be affected by the expressive behaviors of other members during group conversations. Also, we hypothesize that people have a higher probability of feeling specific emotions under the related emotional atmosphere. Therefore, we design an ad-hoc technique by simply summing up the Self-assessed emotional annotations of all group members as the group emotional atmosphere (climate) to help the model predict individuals’ emotions. We propose a Multi-modal Multi-label Emotion based on Transformer BLSTM at Group Emotional Atmosphere Network (MMETBGEAN) that explicitly considers individual changes and dynamic interaction via physiological and acoustic features during a group conversation integrates group emotional atmosphere information for recognizing individuals’ multi-label emotions. We assess the proposed framework on our recently collected extensive Mandarin Chinese collective task group database, NTUBA. The results show that the method outperforms the existing approaches on multi-modal multi-label emotion classification on this database.","PeriodicalId":440830,"journal":{"name":"Companion Publication of the 2021 International Conference on Multimodal Interaction","volume":"10 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128031024","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"When a Voice Assistant Asks for Feedback: An Empirical Study on Customer Experience with A/B Testing and Causal Inference Methods","authors":"Yuqi Deng, Sudeeksha Murari","doi":"10.1145/3461615.3485403","DOIUrl":"https://doi.org/10.1145/3461615.3485403","url":null,"abstract":"Intelligent Voice Assistant (IVA) systems, such as Alexa, Google Assistant and Siri, allow us to interact with them using just the voice commands. IVA systems can seek voice feedback directly from the customers, right after an interaction by simply asking a question such as “did that answer your question?”. We refer to these IVA elicited feedbacks as crowdsourced voice feedback (CVF). In this paper, we look to understand the customer experience (CX) during interactions with an IVA that explicitly seeks feedback. We attempt to quantify the CX of providing feedback, identify the driving factors of CX and offer insights into improving CX with the drivers identified. With an A/B test, we collected data from a leading IVA system and found that feedback elicitations did not impair CX in general. To identify drivers of CX, we performed causal inference with Double Machine Learning. Causal inference teases apart multiple confounding factors and avoids CX risks in experimentation of certain variables. We identified multiple CX drivers including elicitation timing and frequency, which can be useful in establishing guardrails for a CVF system. Our results imply opportunities of CVF systems, and we suggest design specifics that can be leveraged for such feedback collection mechanisms.","PeriodicalId":440830,"journal":{"name":"Companion Publication of the 2021 International Conference on Multimodal Interaction","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128271534","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}