{"title":"Mouse Activity as an Indicator of Interestingness in Video","authors":"Gloria Zen, Paloma de Juan, Yale Song, A. Jaimes","doi":"10.1145/2911996.2912005","DOIUrl":"https://doi.org/10.1145/2911996.2912005","url":null,"abstract":"Automatic detection of interesting moments in video has many real-world applications such as video summarization and efficient online video browsing. In this paper, we present a lightweight and scalable solution to this problem based on user mouse activity while watching video. Unlike previous approaches that analyze video content to infer the interestingness, we leverage the implicit user feedback obtained from thousands of online video watching sessions. This makes our method computationally efficient and scalable to billions of videos. Most importantly, our approach can handle a variety of video genres because we make no assumption on what constitutes interestingness: we let the crowd tell us through their mouse activity. By analyzing 106,212 user sessions collected from a popular online video website, we show that mouse activity is highly indicative of interestingness, and that our approach has competitive performance to several state-of-the-art methods.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"8 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125605595","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Emotion Recognition from EEG Signals Enhanced by User's Profile","authors":"T. Chen, Shangfei Wang, Zhen Gao, Chongliang Wu","doi":"10.1145/2911996.2912050","DOIUrl":"https://doi.org/10.1145/2911996.2912050","url":null,"abstract":"The main stream of current research of emotion recognition from electroencephalogram (EEG) signals infers users' emotion directly from EEG signals, without considering users' profile, which carries crucial information for emotion recognition. To address this, we propose a novel approach of emotion recognition from EEG signals enhanced by users' profile, which is only required during training. Specifically, we propose to use discriminative Restricted Boltzmann Machine (DRBM) to capture the inherent relationships among users' profile, EEG signals and emotion labels. During training, the proposed RBM classifier learns the mapping function from EEG signals and users' profile to emotion labels through discriminative learning algorithm. During testing, users' emotion can be recognized from EEG signals through marginalizing over users' profile, since RBM is a generative model. Experimental results on three benchmark databases, i.e. the DEAP database, the MAHNOB-HCI database and the USTC-ERVS database, demonstrate the effectiveness of the proposed approach in leveraging users' profile for enhancing emotion recognition from EEG signals and the superior emotion recognition performance of the proposed approach over existing approaches.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"13 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132319246","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Emmanouil Schinas, S. Papadopoulos, Georgios Petkos, Y. Kompatsiaris, P. Mitkas
{"title":"Multimodal Event Detection and Summarization in Large Scale Image Collections","authors":"Emmanouil Schinas, S. Papadopoulos, Georgios Petkos, Y. Kompatsiaris, P. Mitkas","doi":"10.1145/2911996.2912031","DOIUrl":"https://doi.org/10.1145/2911996.2912031","url":null,"abstract":"This paper describes a multimodal graph-based approach to address the problem of event detection and summarization in large scale image collections. A first version of our system was presented in the Yahoo-Flickr Event Summarization Challenge of ACM Multimedia 2015 [6]. The objective of the approach is to automatically detect events within millions of photos and summarizing them efficiently for user consumption. The presented approach uses a moving time window over the collection of multimedia items to build a same-event image graph and applies graph clustering to detect events. In addition, it makes use of a graph-based diversity-oriented ranking algorithm to summarize instances of the detected events. A demo of the system is online at: http://mklab.iti.gr/acmmm2015-gc/.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"67 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116629432","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A Computational Approach to Finding Facial Patterns of a Babyface","authors":"Zi-Yi Ke, Mei-Chen Yeh","doi":"10.1145/2911996.2912042","DOIUrl":"https://doi.org/10.1145/2911996.2912042","url":null,"abstract":"Facial babyishness has a strong impact on social perceptions and interactions; however, the components constituting a babyface remain unclear. In this paper, we present a computational approach for identifying important but less apparent facial patterns of a babyface, using voluminous face images on the web. The proposed approach is built upon computationally efficient data mining techniques. A new image set with ground truth data collected from users and an evaluation approach based on age estimation are presented in the experiment. The results show that the mined patterns are effective for understanding and determining babyfaces. The findings of this study should provide information for future investigations on the prediction and analysis of trait impressions using the patterns.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115116074","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Xi Wang, Zhenfeng Sun, Wenqiang Zhang, Yu Zhou, Yu-Gang Jiang
{"title":"Matching User Photos to Online Products with Robust Deep Features","authors":"Xi Wang, Zhenfeng Sun, Wenqiang Zhang, Yu Zhou, Yu-Gang Jiang","doi":"10.1145/2911996.2912002","DOIUrl":"https://doi.org/10.1145/2911996.2912002","url":null,"abstract":"This paper focuses on a practically very important problem of matching a real-world product photo to exactly the same item(s) in online shopping sites. The task is extremely challenging because the user photos (i.e., the queries in this scenario) are often captured in uncontrolled environments, while the product images in online shops are mostly taken by professionals with clean backgrounds and perfect lighting conditions. To tackle the problem, we study deep network architectures and training schemes, with the goal of learning a robust deep feature representation that is able to bridge the domain gap between the user photos and the online product images. Our contributions are two-fold. First, we propose an alternative of the popular contrastive loss used in siamese deep networks, namely robust contrastive loss, where we \"relax\" the penalty on positive pairs to alleviate over-fitting. Second, a multi-task fine-tuning approach is introduced to learn a better feature representation, which not only incorporates knowledge from the provided training photo pairs, but also explores additional information from the large ImageNet dataset to regularize the fine-tuning procedure. Experiments on two challenging real-world datasets demonstrate that both the robust contrastive loss and the multi-task fine-tuning approach are effective, leading to very promising results with a time cost suitable for real-time retrieval.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116256194","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Multimodal Analysis of User-Generated Content in Support of Social Media Applications","authors":"R. Shah","doi":"10.1145/2911996.2912032","DOIUrl":"https://doi.org/10.1145/2911996.2912032","url":null,"abstract":"The number of user-generated multimedia content (UGC) online has increased rapidly in recent years due to the ubiquitous availability of smartphones, cameras, and affordable network infrastructures. Thus, it attracts companies to provide diverse multimedia-related services such as preference-aware multimedia recommendations, multimedia-based e--learning, and event summarization from a large collection of multimedia content. However, a real-world UGC is complex and extracting semantics from only multimedia content is difficult because suitable concepts may be exhibited in different representations. Modern devices capture contextual information in conjunction with a multimedia content, which greatly facilitates in the semantics understanding of the multimedia content. Thus, it is beneficial to analyse UGC from multiple modalities such as multimedia content and contextual information (eg., spatial and temporal information). This doctoral research studies the multimodal analysis of UGC in support of above-mentioned social media problems. We present our proposed approaches, results, and works in progress on these problems.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124001927","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Prithwi Raj Chakraborty, D. Tjondronegoro, Ligang Zhang, V. Chandran
{"title":"Automatic Identification of Sports Video Highlights using Viewer Interest Features","authors":"Prithwi Raj Chakraborty, D. Tjondronegoro, Ligang Zhang, V. Chandran","doi":"10.1145/2911996.2912011","DOIUrl":"https://doi.org/10.1145/2911996.2912011","url":null,"abstract":"Classification of viewer interest using facial expression and heart rate facilitates automatic identification of interest evoking video segments. Sports video is suitable for testing the effectiveness of such a system as it has structured segments with distinguishable highlight events. Previous work has not investigated the differences in viewer interest characteristics from one sports type to another, which is crucial for appropriate classification methodology. Thus, it is still unclear whether a universal classification model can be used for analyzing viewer interest for all types of sports. This paper addresses this gap by demonstrating a significant difference (p < 0.05) in the distributions of viewer interest data in soccer compared to tennis. Based on this finding, this paper proposes an adoption of Gaussian mixture models (GMM) to integrate sports-specific and sports-independent approaches for identifying video segments, which would be of potential interest to individual viewers. The approaches achieve 52% to 64% accuracy, demonstrating that sports-specific approach gets better performance.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"32 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123954059","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Adding Chinese Captions to Images","authors":"Xirong Li, Weiyu Lan, Jianfeng Dong, Hailong Liu","doi":"10.1145/2911996.2912049","DOIUrl":"https://doi.org/10.1145/2911996.2912049","url":null,"abstract":"This paper extends research on automated image captioning in the dimension of language, studying how to generate Chinese sentence descriptions for unlabeled images. To evaluate image captioning in this novel context, we present Flickr8k-CN, a bilingual extension of the popular Flickr8k set. The new multimedia dataset can be used to quantitatively assess the performance of Chinese captioning and English-Chinese machine translation. The possibility of re-using existing English data and models via machine translation is investigated. Our study reveals to some extent that a computer can master two distinct languages, English and Chinese, at a similar level for describing the visual world. Data is publicly available at http://tinyurl.com/flickr8kcn","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"45 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128538152","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Homemade TS-Net for Automatic Face Recognition","authors":"Shilun Lin, Zhicheng Zhao, Fei Su","doi":"10.1145/2911996.2911999","DOIUrl":"https://doi.org/10.1145/2911996.2911999","url":null,"abstract":"Inspired by how human being accomplishes face recognition task, a new architecture, called transfer and specialized net (TS-Net) is proposed in this paper, which fuses the general and specialized knowledge by combining a Transfer FaceNet and a Specialized FaceNet. The former is obtained by fine-tuning the pre-trained GoogleNet to transfer object-recognition knowledge to face recognition, and the latter is trained on global and local face patches to provide the discriminative specialized knowledge for face recognition. The final face representation is formed by fusing the features from both FaceNets. The advantages of our proposed architecture come from that: (i) By explicitly assigning different learning rates to different layers we successfully transfer the well-trained GoogleNet from object recognition task to a distinctly different task - face recognition; (ii) We construct the Specialized FaceNet with 6 simple networks to imitate the capture of featured-based and configural information in human vision process; (iii) Both Transfer FaceNet and Specialized FaceNets can be trained with a relatively small amount of training data (about 0.4 million samples) and a low configuration hardware (for example, a Titan-Z GPU). Experimental results show that TS-Net achieves competitive performance on both LFW and CASIA-Webface datasets. Also, it is promising that only slight dropping is found on verification and identification accuracy when 300 dimensional binary face representations are applied with Cosine distance as measure, which is implemental to develop practical human face retrieval and recognition system.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"44 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126807321","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Qing Li, Zhaofan Qiu, Ting Yao, Tao Mei, Y. Rui, Jiebo Luo
{"title":"Action Recognition by Learning Deep Multi-Granular Spatio-Temporal Video Representation","authors":"Qing Li, Zhaofan Qiu, Ting Yao, Tao Mei, Y. Rui, Jiebo Luo","doi":"10.1145/2911996.2912001","DOIUrl":"https://doi.org/10.1145/2911996.2912001","url":null,"abstract":"Recognizing actions in videos is a challenging task as video is an information-intensive media with complex variations. Most existing methods have treated video as a flat data sequence while ignoring the intrinsic hierarchical structure of the video content. In particular, an action may span different granularities in this hierarchy including, from small to large, a single frame, consecutive frames (motion), a short clip, and the entire video. In this paper, we present a novel framework to boost action recognition by learning a deep spatio-temporal video representation at hierarchical multi-granularity. Specifically, we model each granularity as a single stream by 2D (for frame and motion streams) or 3D (for clip and video streams) convolutional neural networks (CNNs). The framework therefore consists of multi-stream 2D or 3D CNNs to learn both the spatial and temporal representations. Furthermore, we employ the Long Short-Term Memory (LSTM) networks on the frame, motion, and clip streams to exploit long-term temporal dynamics. With a softmax layer on the top of each stream, the classification scores can be predicted from all the streams, followed by a novel fusion scheme based on the multi-granular score distribution. Our networks are learned in an end-to-end fashion. On two video action benchmarks of UCF101 and HMDB51, our framework achieves promising performance compared with the state-of-the-art.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134039460","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}