Proceedings of the 2022 International Conference on Multimodal Interaction最新文献

筛选
英文 中文
A Spatio-temporal Learning for Music Conditioned Dance Generation 音乐条件舞蹈生成的时空学习
Proceedings of the 2022 International Conference on Multimodal Interaction Pub Date : 2022-11-07 DOI: 10.1145/3536221.3556618
Li Zhou, Yan Luo
{"title":"A Spatio-temporal Learning for Music Conditioned Dance Generation","authors":"Li Zhou, Yan Luo","doi":"10.1145/3536221.3556618","DOIUrl":"https://doi.org/10.1145/3536221.3556618","url":null,"abstract":"The music-conditioned dance generation, i.e., dancing to music, is a usage scenario of multi-modality human motion synthesis. Typically, it is a challenge to choreograph continuous motions coinciding with the melody and rhythm of the music. This paper proposes a position-wise encoding-decoding framework for spatio-temporal learning of motions and long-term skeleton-based dance generation oriented on music. Given the positional embedding of the frames in 1-minute video clips, firstly, we modularize a regional attention-based feed-forward mechanism to encode the music features. Secondly, based on the skeleton of each frame and the joint trajectories across motion frames, we formalize a graph topology to represent each dance sequence’s spatial and temporal knowledge. Specifically, we propose a graph convolutional network (GCN) based blocks to process long-term dependencies of motions and leverage the spatial and temporal features. Both music and motion paths are learned fully in positional embedding schemes and constructed by repeating the corresponding blocks. Finally, as the task of dance generation is inherently the consistency between music and motions, we proposed a cross-modality feature fusion for multimodal interaction and music-conditioned dance generation. Experimental results demonstrate that our method outperforms state-of-art methods in motion quality and motion-music correlation metrics.","PeriodicalId":237261,"journal":{"name":"Proceedings of the 2022 International Conference on Multimodal Interaction","volume":"29 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-11-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134235634","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
A cognitive knowledge-based system for hair and makeup recommendation based on facial features classification 基于面部特征分类的发型和妆容推荐认知知识系统
Proceedings of the 2022 International Conference on Multimodal Interaction Pub Date : 2022-11-07 DOI: 10.1145/3536221.3556594
Juhyun Lee, Joosun Yum, Marvin Lee, Ji-Hyun Lee
{"title":"A cognitive knowledge-based system for hair and makeup recommendation based on facial features classification","authors":"Juhyun Lee, Joosun Yum, Marvin Lee, Ji-Hyun Lee","doi":"10.1145/3536221.3556594","DOIUrl":"https://doi.org/10.1145/3536221.3556594","url":null,"abstract":"This paper aims at building a knowledge-based system of smart mirrors for the cosmetic industry. In order to better understand hair and cosmetic experts, we first conduct interviews for knowledge acquisition. Then, we obtain insights from each category of answers collected from expert interviews. To design this knowledge-based system, we define concepts, main tasks and subtasks in order to extract rules to design the system. This system can suggest hairstyles and make-up colors by considering users’ face shapes and their personal colors.","PeriodicalId":237261,"journal":{"name":"Proceedings of the 2022 International Conference on Multimodal Interaction","volume":"11 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-11-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114362403","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
Does Audio help in deep Audio-Visual Saliency prediction models? 音频对深度视听显著性预测模型有帮助吗?
Proceedings of the 2022 International Conference on Multimodal Interaction Pub Date : 2022-11-07 DOI: 10.1145/3536221.3556625
Ritvik Agrawal, Shreyank Jyoti, Rohit Girmaji, S. Sivaprasad, Vineet Gandhi
{"title":"Does Audio help in deep Audio-Visual Saliency prediction models?","authors":"Ritvik Agrawal, Shreyank Jyoti, Rohit Girmaji, S. Sivaprasad, Vineet Gandhi","doi":"10.1145/3536221.3556625","DOIUrl":"https://doi.org/10.1145/3536221.3556625","url":null,"abstract":"Despite existing works of Audio-Visual Saliency Prediction (AVSP) models claiming to achieve promising results by fusing audio modality over visual-only models, these models fail to leverage audio information. In this paper, we investigate the relevance of audio cues in conjunction with the visual ones and conduct extensive analysis by employing well-established audio modules and fusion techniques from diverse correlated audio-visual tasks. Our analysis on ten diverse saliency datasets suggests that none of the methods worked for incorporating audio. Furthermore, we bring to light, why AVSP models show a gain in performance over visual-only models, though the audio branch is agnostic at inference. Our work questions the role of audio in current deep AVSP models and motivates the community to a clear avenue for reconsideration of the complex architectures by demonstrating that simpler alternatives work equally well.","PeriodicalId":237261,"journal":{"name":"Proceedings of the 2022 International Conference on Multimodal Interaction","volume":"672 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-11-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115122193","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
A Deep Dive Into Neural Synchrony Evaluation for Audio-visual Translation 视听翻译的神经同步性评价研究
Proceedings of the 2022 International Conference on Multimodal Interaction Pub Date : 2022-11-07 DOI: 10.1145/3536221.3556621
Shravan Nayak, Christian Schuler, Debjoy Saha, Timo Baumann
{"title":"A Deep Dive Into Neural Synchrony Evaluation for Audio-visual Translation","authors":"Shravan Nayak, Christian Schuler, Debjoy Saha, Timo Baumann","doi":"10.1145/3536221.3556621","DOIUrl":"https://doi.org/10.1145/3536221.3556621","url":null,"abstract":"We present a comprehensive analysis of the neural audio-visual synchrony evaluation tool SyncNet. We assess the agreement of SyncNet scores vis-a-vis human perception and whether we can use these as a reliable metric for evaluating audio-visual lip-synchrony in generation tasks with no ground truth reference audio-video pair. We further look into the underlying elements in audio and video which vitally affect synchrony using interpretable explanations from SyncNet predictions and analyse its susceptibility by introducing adversarial noise. SyncNet has been used in numerous papers on visually-grounded text-to-speech for scenarios such as dubbing. We focus on this scenario which features many local asynchronies (something that SyncNet isn’t made for).","PeriodicalId":237261,"journal":{"name":"Proceedings of the 2022 International Conference on Multimodal Interaction","volume":"14 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-11-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117294436","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
GestureMaster: Graph-based Speech-driven Gesture Generation GestureMaster:基于图形的语音驱动手势生成
Proceedings of the 2022 International Conference on Multimodal Interaction Pub Date : 2022-11-07 DOI: 10.1145/3536221.3558063
Chi Zhou, Tengyue Bian, Kang Chen
{"title":"GestureMaster: Graph-based Speech-driven Gesture Generation","authors":"Chi Zhou, Tengyue Bian, Kang Chen","doi":"10.1145/3536221.3558063","DOIUrl":"https://doi.org/10.1145/3536221.3558063","url":null,"abstract":"This paper describes the GestureMaster entry to the GENEA (Generation and Evaluation of Non-verbal Behaviour for Embodied Agents) Challenge 2022. Given speech audio and text transcriptions, GestureMaster can automatically generate a high-quality gesture sequence to accompany the input audio and text transcriptions in terms of style and rhythm. GestureMaster system is based on the recent ChoreoMaster publication[12]. ChoreoMaster can generate dance motion given a piece of music. We make some adjustments to ChoreoMaster to suit for the speech-driven gesture generation task. We are pleased to see that among the participating systems, our entry attained the highest median score in the human-likeness evaluation. In the appropriateness evaluation, we ranked first in upper-body study and second in full-body study.","PeriodicalId":237261,"journal":{"name":"Proceedings of the 2022 International Conference on Multimodal Interaction","volume":"34 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-11-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123409088","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 16
Cognitive Workload Assessment via Eye Gaze and EEG in an Interactive Multi-Modal Driving Task 基于眼注视和脑电的交互式多模态驾驶任务认知负荷评估
Proceedings of the 2022 International Conference on Multimodal Interaction Pub Date : 2022-11-07 DOI: 10.1145/3536221.3556610
Ayca Aygun, Boyang Lyu, Thuan Q. Nguyen, Z. Haga, Shuchin Aeron, matthias. scheutz
{"title":"Cognitive Workload Assessment via Eye Gaze and EEG in an Interactive Multi-Modal Driving Task","authors":"Ayca Aygun, Boyang Lyu, Thuan Q. Nguyen, Z. Haga, Shuchin Aeron, matthias. scheutz","doi":"10.1145/3536221.3556610","DOIUrl":"https://doi.org/10.1145/3536221.3556610","url":null,"abstract":"Assessing the cognitive workload of human interactants in mixed-initiative teams is a critical capability for autonomous interactive systems to enable adaptations that improve team performance. Yet, it is still unclear, due to diverging evidence, which sensing modality might work best for the determination of human workload. In this paper, we report results from an empirical study that was designed to answer this question by collecting eye gaze and electroencephalogram (EEG) data from human subjects performing an interactive multi-modal driving task. Different levels of cognitive workload were generated by introducing secondary tasks like dialogue, braking events, and tactile stimulation in the course of driving. Our results show that pupil diameter is a more reliable indicator for workload prediction than EEG. And more importantly, none of the five different machine learning models combining the extracted EEG and pupil diameter features were able to show any improvement in workload classification over eye gaze alone, suggesting that eye gaze is a sufficient modality for assessing human cognitive workload in interactive, multi-modal, multi-task settings.","PeriodicalId":237261,"journal":{"name":"Proceedings of the 2022 International Conference on Multimodal Interaction","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-11-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126801056","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 3
Multimodal classification of interruptions in humans’ interaction 人类互动中断的多模态分类
Proceedings of the 2022 International Conference on Multimodal Interaction Pub Date : 2022-11-07 DOI: 10.1145/3536221.3556604
Liu Yang, C. Achard, C. Pelachaud
{"title":"Multimodal classification of interruptions in humans’ interaction","authors":"Liu Yang, C. Achard, C. Pelachaud","doi":"10.1145/3536221.3556604","DOIUrl":"https://doi.org/10.1145/3536221.3556604","url":null,"abstract":"During an interaction interruptions occur frequently. Interruptions may arise to fulfill different goals such as changing the topic of conversation abruptly, asking for clarification, completing the current speaker’s turn. Interruptions may be cooperative or competitive depending on the interrupter’s intention. Our main goal is to endow a Socially Interactive Agent with the capacity to handle user interruptions in dyadic interaction. It requires the agent to detect an interruption and recognize its type (cooperative/competitive), and then to plan its behaviours to respond appropriately. As a first step towards this goal, we developed a multimodal classification model using acoustic features, facial expression, head movement, and gaze direction from both, the interrupter and the interruptee. The classification model learns from the sequential information to automatically identify interruptions type. We also present studies we conducted to measure the shortest delay needed (0.6s) for our classification model to identify interruption types with a high classification accuracy (81%). On average, most interruption overlaps last longer than 0.6s, so a Socially Interactive Agent has time to detect and recognize an interruption type and can respond in a timely manner to its human interlocutor’s interruption.","PeriodicalId":237261,"journal":{"name":"Proceedings of the 2022 International Conference on Multimodal Interaction","volume":"60 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-11-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121017010","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
DynaTags: Low-Cost Fiducial Marker Mechanisms 低成本基准标记机制
Proceedings of the 2022 International Conference on Multimodal Interaction Pub Date : 2022-11-07 DOI: 10.1145/3536221.3556591
C. Scheirer, Chris Harrison
{"title":"DynaTags: Low-Cost Fiducial Marker Mechanisms","authors":"C. Scheirer, Chris Harrison","doi":"10.1145/3536221.3556591","DOIUrl":"https://doi.org/10.1145/3536221.3556591","url":null,"abstract":"Printed fiducial markers are inexpensive, easy to deploy, robust and deservedly popular. However, their data payload is also static, unable to express any state beyond being present. For this reason, more complex electronic tagging technologies exist, which can sense and change state, but either require special equipment to read or are orders of magnitude more expensive than printed markers. In this work, we explore an approach between these two extremes: one that retains the simple, low-cost nature of printed markers, yet has some of the expressive capabilities of dynamic tags. Our “DynaTags” are simple mechanisms constructed from paper that express multiple payloads, allowing practitioners and researchers to create new and compelling physical-digital experiences. We describe a library of 23 mechanisms that can be read by standard smartphone reader apps. Through a series of demo applications (augmenting reality through e.g., sounds, environmental lighting and graphics) we show how our tags can bring new interactivity to previously static experiences.","PeriodicalId":237261,"journal":{"name":"Proceedings of the 2022 International Conference on Multimodal Interaction","volume":"38 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-11-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126987977","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 3
EdgeSelect: Smartwatch Data Interaction with Minimal Screen Occlusion EdgeSelect:最小屏幕遮挡的智能手表数据交互
Proceedings of the 2022 International Conference on Multimodal Interaction Pub Date : 2022-11-07 DOI: 10.1145/3536221.3556586
Ali Neshati, Aaro Salo, Shariff A. M. Faleel, Ziming Li, Hai-Ning Liang, C. Latulipe, Pourang Irani
{"title":"EdgeSelect: Smartwatch Data Interaction with Minimal Screen Occlusion","authors":"Ali Neshati, Aaro Salo, Shariff A. M. Faleel, Ziming Li, Hai-Ning Liang, C. Latulipe, Pourang Irani","doi":"10.1145/3536221.3556586","DOIUrl":"https://doi.org/10.1145/3536221.3556586","url":null,"abstract":"We present EdgeSelect, a linear target selection interaction technique that utilizes a small portion of the smartwatch display, explicitly designed to mitigate the ‘fat finger’ and screen occlusion problems, two of the most common and well-known challenges when interacting with small displays. To design our technique, we first conducted a user study to answer which segments of the smartwatch display have the least screen occlusion while users are interacting with it. We use results from the first experiment to introduce EdgeSelect, a three-layer non-linear interaction technique, which can be used to interact with multiple co-adjacent graphs on the smartwatch by using a region that is the least prone to finger occlusion. In a second experiment, we explore the density limits of the targets possible with EdgeSelect. Finally, we demonstrate the generalizability of EdgeSelect to interact with various types of content.","PeriodicalId":237261,"journal":{"name":"Proceedings of the 2022 International Conference on Multimodal Interaction","volume":"84 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-11-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133268836","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
Keep in Touch: Combining Touch Interaction with Thumb-to-Finger µGestures for People with Visual Impairment 保持联系:结合触摸互动与拇指到手指的微手势为视障人士
Proceedings of the 2022 International Conference on Multimodal Interaction Pub Date : 2022-11-07 DOI: 10.1145/3536221.3556589
Gauthier Robert Jean Faisandaz, Alix Goguey, C. Jouffrais, L. Nigay
{"title":"Keep in Touch: Combining Touch Interaction with Thumb-to-Finger µGestures for People with Visual Impairment","authors":"Gauthier Robert Jean Faisandaz, Alix Goguey, C. Jouffrais, L. Nigay","doi":"10.1145/3536221.3556589","DOIUrl":"https://doi.org/10.1145/3536221.3556589","url":null,"abstract":"We present a set of 8 thumb-to-finger microgestures (TTF µGestures) that can be used as an additional modality to enrich touch interaction in eyes-free situations. TTF µGestures possess characteristics especially suited for people with visual impairment (PVI). They have never been studied specifically for PVI to improve accessibility of touchscreen devices. We studied a set of 33 common TTF µGestures to determine which are feasible and usable without seeing while the index is touching a surface. We found that the constrained position of the hand and the absence of vision prevent participants from being able to efficiently target a specific phalanx. Thus, we propose a set of 8 TTF µGestures (6 taps, 2 swipes) balancing resiliency (i.e., low error-rate) and expressivity (i.e., number of possible inputs): as a dimension combined with the touch modality, it would realistically multiply the touch command space by eight. Within our set of 8 TTF µGestures, we chose a subset of 4 µGestures (2 taps and 2 swipes) and implemented an exploration scenario of an audio-tactile map with a raised-line overlay on a touchscreen and tested it with 7 PVI. Their feedback was positive on the potential benefits of TTF µGestures in enhancing the touch modality and supporting PVI interaction with touchscreen devices.","PeriodicalId":237261,"journal":{"name":"Proceedings of the 2022 International Conference on Multimodal Interaction","volume":"191 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-11-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133503115","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
相关产品
×
本文献相关产品
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信
小红书