{"title":"An Audio-Visual Method for Room Boundary Estimation and Material Recognition","authors":"Luca Remaggi, Hansung Kim, P. Jackson, A. Hilton","doi":"10.1145/3264869.3264876","DOIUrl":"https://doi.org/10.1145/3264869.3264876","url":null,"abstract":"In applications such as virtual and augmented reality, a plausible and coherent audio-visual reproduction can be achieved by deeply understanding the reference scene acoustics. This requires knowledge of the scene geometry and related materials. In this paper, we present an audio-visual approach for acoustic scene understanding. We propose a novel material recognition algorithm, that exploits information carried by acoustic signals. The acoustic absorption coefficients are selected as features. The training dataset was constructed by combining information available in the literature, and additional labeled data that we recorded in a small room having short reverberation time (RT60). Classic machine learning methods are used to validate the model, by employing data recorded in five rooms, having different sizes and RT60s. The estimated materials are utilized to label room boundaries, reconstructed by a vision-based method. Results show 89% and 80% agreement between the estimated and reference room volumes and materials, respectively.","PeriodicalId":378296,"journal":{"name":"Proceedings of the 2018 Workshop on Audio-Visual Scene Understanding for Immersive Multimedia","volume":"23 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126942489","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Proceedings of the 2018 Workshop on Audio-Visual Scene Understanding for Immersive Multimedia","authors":"","doi":"10.1145/3264869","DOIUrl":"https://doi.org/10.1145/3264869","url":null,"abstract":"","PeriodicalId":378296,"journal":{"name":"Proceedings of the 2018 Workshop on Audio-Visual Scene Understanding for Immersive Multimedia","volume":"59 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127046342","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Multimodal Fusion Strategies: Human vs. Machine","authors":"Hanseok Ko","doi":"10.1145/3264869.3264870","DOIUrl":"https://doi.org/10.1145/3264869.3264870","url":null,"abstract":"Two-hour movie or a short movie clip as its subset is intended to capture and present a meaningful (or significant) story in video to be recognized and understood by human audience. What if we substitute the task of human audience with that of an intelligent machine or robot capable of capturing and processing the semantic information in terms of audio and video cues contained in the video? By using both auditory and visual means, human brain processes the audio (sound, speech) and video (background image scene, moving video objects, written characters) modalities to extract the spatial and temporal semantic information, that are contextually complementary and robust. Smart machines equipped with audiovisual multisensors (e.g. CCTV equipped with cameras and microphones) should be capable of achieving the same task. An appropriate fusion strategy combining the audio and visual information would be a key component in developing such artificial general intelligent (AGI) systems. This talk reviews the challenges of current video analytics schemes and explores various sensor fusion techniques [1, 2, 3, 4] to combine the audio-visual information cues for video content analytics task.","PeriodicalId":378296,"journal":{"name":"Proceedings of the 2018 Workshop on Audio-Visual Scene Understanding for Immersive Multimedia","volume":"30 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122142376","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
G. Llorach, G. Grimm, Maartje M. E. Hendrikse, V. Hohmann
{"title":"Towards Realistic Immersive Audiovisual Simulations for Hearing Research: Capture, Virtual Scenes and Reproduction","authors":"G. Llorach, G. Grimm, Maartje M. E. Hendrikse, V. Hohmann","doi":"10.1145/3264869.3264874","DOIUrl":"https://doi.org/10.1145/3264869.3264874","url":null,"abstract":"Most current hearing research laboratories and hearing aid evaluation setups are not sufficient to simulate real-life situations and to evaluate future generations of hearing aids that might include gaze information and brain signals. Thus, new methodologies and technologies might need to be implemented in hearing laboratories and clinics in order to generate audiovisual realistic testing environments. The aim of this work is to provide a comprehensive review of the current available approaches and future directions to create audiovisual realistic immersive simulations for hearing research. Additionally, we present the technologies and use cases of our laboratory, as well as the pros and cons of such technologies: From creating 3D virtual simulations with computer graphics and virtual acoustic simulations, to 360º videos and Ambisonic recordings.","PeriodicalId":378296,"journal":{"name":"Proceedings of the 2018 Workshop on Audio-Visual Scene Understanding for Immersive Multimedia","volume":"15 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132315446","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Audio-Visual Attention Networks for Emotion Recognition","authors":"Jiyoung Lee, Sunok Kim, Seungryong Kim, K. Sohn","doi":"10.1145/3264869.3264873","DOIUrl":"https://doi.org/10.1145/3264869.3264873","url":null,"abstract":"We present a spatiotemporal attention based multimodal deep neu- ral networks for dimensional emotion recognition in multimodal audio-visual video sequence. To learn the temporal attention that discriminatively focuses on emotional sailient parts within speech audios, we formulate the temporal attention network using deep neural networks (DNNs). In addition, to learn the spatiotemporal attention that selectively focuses on emotional sailient parts within facial videos, the spatiotemporal encoder-decoder network is formu- lated using Convolutional LSTM (ConvLSTM) modules, and learned implicitly without any pixel-level annotations. By leveraging the spatiotemporal attention, the 3D convolutional neural networks (3D-CNNs) is also formulated to robustly recognize the dimensional emotion in facial videos. Furthermore, to exploit multimodal infor- mation, we fuse the audio and video features to emotion regression model. The experimental results show that our method can achieve the state-of-the-art results in dimensional emotion recognition with the highest concordance correlation coefficient (CCC) on AV+EC 2017 dataset.","PeriodicalId":378296,"journal":{"name":"Proceedings of the 2018 Workshop on Audio-Visual Scene Understanding for Immersive Multimedia","volume":"47 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126942688","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Session details: Oral Session 1","authors":"Bumsub Ham","doi":"10.1145/3286581","DOIUrl":"https://doi.org/10.1145/3286581","url":null,"abstract":"","PeriodicalId":378296,"journal":{"name":"Proceedings of the 2018 Workshop on Audio-Visual Scene Understanding for Immersive Multimedia","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129189690","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Session details: Keynote Talk 2","authors":"Hansung Kim","doi":"10.1145/3286582","DOIUrl":"https://doi.org/10.1145/3286582","url":null,"abstract":"","PeriodicalId":378296,"journal":{"name":"Proceedings of the 2018 Workshop on Audio-Visual Scene Understanding for Immersive Multimedia","volume":"26 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121233133","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Spatial Audio on the Web - Create, Compress, and Render","authors":"J. Skoglund","doi":"10.1145/3264869.3264871","DOIUrl":"https://doi.org/10.1145/3264869.3264871","url":null,"abstract":"The recent surge of VR and AR has spawned an interest in spatial audio beyond its traditional delivery over loudspeakers in, e.g., home theater environments, to headphone delivery over, e.g., mobile devices. In this talk we'll discuss a web-based approach to spatial audio. It will cover creating real-time spatial audio directly in the browser, data compression of the audio for immersive media, and efficient binaural rendering.","PeriodicalId":378296,"journal":{"name":"Proceedings of the 2018 Workshop on Audio-Visual Scene Understanding for Immersive Multimedia","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115975661","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
O. Takeuchi, Hidehiko Shishido, Y. Kameda, Hansung Kim, I. Kitahara
{"title":"Generation Method for Immersive Bullet-Time Video Using an Omnidirectional Camera in VR Platform","authors":"O. Takeuchi, Hidehiko Shishido, Y. Kameda, Hansung Kim, I. Kitahara","doi":"10.1145/3264869.3264872","DOIUrl":"https://doi.org/10.1145/3264869.3264872","url":null,"abstract":"This paper proposes a generation method of immersive bullet-time video that continuously switches the images captured by multi-viewpoint omnidirectional cameras arranged around the subject. In ordinary bullet-time processing, it is possible to observe a point of interest (POI) at the same screen position by applying projective transformation to captured multi-viewpoint images. However, the observable area is limited by the field of view of the capturing cameras. Thus, a blank region is added to the displayed image, depending on the spatial relationship between the POI and the capturing camera. This seriously harms image quality (i.e., immersiveness). We solve this problem by applying omnidirectional cameras to bullet-time video production. Furthermore, by using the virtual reality platform for calibration of multi-viewpoint omnidirectional cameras and display of bullet-time video, fast and simple processing can be realised.","PeriodicalId":378296,"journal":{"name":"Proceedings of the 2018 Workshop on Audio-Visual Scene Understanding for Immersive Multimedia","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128214934","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Session details: Keynote Talk 1","authors":"A. Hilton","doi":"10.1145/3286580","DOIUrl":"https://doi.org/10.1145/3286580","url":null,"abstract":"","PeriodicalId":378296,"journal":{"name":"Proceedings of the 2018 Workshop on Audio-Visual Scene Understanding for Immersive Multimedia","volume":"5 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2018-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116916823","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}