Zhiwei Fang, J. Liu, Yuhang Wang, Yong Li, Hang Song, Jinhui Tang, Hanqing Lu
{"title":"Object-aware Deep Network for Commodity Image Retrieval","authors":"Zhiwei Fang, J. Liu, Yuhang Wang, Yong Li, Hang Song, Jinhui Tang, Hanqing Lu","doi":"10.1145/2911996.2912027","DOIUrl":"https://doi.org/10.1145/2911996.2912027","url":null,"abstract":"Recent years, with the development of e-commerce and population of mobile phones, image-based commodity retrieval has attracted much attention. This paper proposed a deep framework for commodity image retrieval(CMIR) from the view that they are same designed commodities. Our framework can catch as many design details as possible by exploring object detection and ranking sensitive feature learning, while the former is performed based on Faster R-CNN, and the later is learned with a multi-task Siamese Network. Besides, we refine the processing speed of the framework to make it a live system. Our framework is implemented on an android application based on Client/Server structure model whose server response time is about 150 ms per query.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131066240","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Searching for Audio by Sketching Mental Images of Sound: A Brave New Idea for Audio Retrieval in Creative Music Production","authors":"Peter Knees, Kristina Andersen","doi":"10.1145/2911996.2912021","DOIUrl":"https://doi.org/10.1145/2911996.2912021","url":null,"abstract":"We propose a new paradigm for searching for sound by allowing users to graphically sketch their mental representation of sound as query. By conducting interviews with professional music producers and creators, we find that existing, text-based indexing and retrieval methods based on file names and tags to search for sound material in large collections (e.g., sample databases) do not reflect their mental concepts, which are often rooted in the visual domain and hence are far from their actual needs, work practices, and intuition. As a consequence, when creating new music on the basis of existing sounds, the process of finding these sounds is cumbersome and breaks their work flow by being forced to resort to browsing the collection. Prior work on organizing sound repositories aiming at bridging this conceptual gap between sound and vision builds upon psychological findings (often alluding to synaesthetic phenomena) or makes use of ad-hoc, technology-driven mappings. These methods foremost aim at visualizing the contents of collections or individual sounds and, again, facilitating browsing therein. For the purpose of indexing and querying, such methods have not been applied yet. We argue that the development of a search system that allows for visual queries to audio collections is desired by users and should inform and drive future research in audio retrieval. To explore this notion, we test the idea of a sketch interface with music producers in a semi-structured interview process by making use of a physical non-functional prototype. Based on the outcomes of this study, we propose a conceptual software prototype for visually querying sound repositories using image manipulation metaphors.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133664908","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Multilingual Visual Sentiment Concept Matching","authors":"Nikolaos Pappas, Miriam Redi, Mercan Topkara, Brendan Jou, Hongyi Liu, Tao Chen, Shih-Fu Chang","doi":"10.1145/2911996.2912016","DOIUrl":"https://doi.org/10.1145/2911996.2912016","url":null,"abstract":"The impact of culture in visual emotion perception has recently captured the attention of multimedia research. In this study, we provide powerful computational linguistics tools to explore, retrieve and browse a dataset of 16K multilingual affective visual concepts and 7.3M Flickr images. First, we design an effective crowdsourcing experiment to collect human judgements of sentiment connected to the visual concepts. We then use word embeddings to represent these concepts in a low dimensional vector space, allowing us to expand the meaning around concepts, and thus enabling insight about commonalities and differences among different languages. We compare a variety of concept representations through a novel evaluation task based on the notion of visual semantic relatedness. Based on these representations, we design clustering schemes to group multilingual visual concepts, and evaluate them with novel metrics based on the crowdsourced sentiment annotations as well as visual semantic relatedness. The proposed clustering framework enables us to analyze the full multilingual dataset in-depth and also show an application on a facial data subset, exploring cultural insights of portrait-related affective visual concepts.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"20 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115596385","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jan Zahálka, S. Rudinac, B. Jónsson, D. Koelma, M. Worring
{"title":"Interactive Multimodal Learning on 100 Million Images","authors":"Jan Zahálka, S. Rudinac, B. Jónsson, D. Koelma, M. Worring","doi":"10.1145/2911996.2912062","DOIUrl":"https://doi.org/10.1145/2911996.2912062","url":null,"abstract":"This paper presents Blackthorn, an efficient interactive multimodal learning approach facilitating analysis of multimedia collections of 100 million items on a single high-end workstation. This is achieved by efficient data compression and optimizations to the interactive learning process. The compressed i-I64 data representation costs tens of bytes per item yet preserves most of the visual and textual semantic information. The optimized interactive learning model scores the i-I64-compressed data directly, greatly reducing the computational requirements. The experiments show that Blackthorn is up to 105x faster than the conventional relevance feedback baseline. Blackthorn is shown to vastly outperform the baseline with respect to recall over time. Blackthorn reaches up to 92% of the precision achieved by the baseline, validating the efficacy of the i-I64 representation. On the YFCC100M dataset, Blackthorn performes one complete interaction round in 0.7 seconds. Blackthorn thus opens multimedia collections comprising 100 million items to learning-based analysis in fully interactive time.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"106 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114615965","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Video Description Generation using Audio and Visual Cues","authors":"Qin Jin, Junwei Liang","doi":"10.1145/2911996.2912043","DOIUrl":"https://doi.org/10.1145/2911996.2912043","url":null,"abstract":"The recent advances in image captioning stimulate the research in generating natural language description for visual content, which can be widely applied in many applications such as assisting blind people. Video description generation is a more complex task than image caption. Most works of video description generation focus on visual information in the video. However, audio provides rich information for describing video contents as well. In this paper, we propose to generate video descriptions in natural sentences using both audio and visual cues. We use unified deep neural networks with both convolutional and recurrent structure. Experimental results on the Microsoft Research Video Description (MSVD) corpus prove that fusing audio information greatly improves the video description performance.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125967047","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
S. Battiato, G. Farinella, F. L. M. Milotta, A. Ortis, Luca Addesso, A. Casella, V. D'Amico, Giovanni Torrisi
{"title":"The Social Picture","authors":"S. Battiato, G. Farinella, F. L. M. Milotta, A. Ortis, Luca Addesso, A. Casella, V. D'Amico, Giovanni Torrisi","doi":"10.1145/2911996.2912024","DOIUrl":"https://doi.org/10.1145/2911996.2912024","url":null,"abstract":"We present The Social Picture, a framework to collect and explore huge amount of crowdsourced social images about public events, cultural heritage sites and other customized private events.The Social Picture aims to create social communities of users that contribute to the creation of image collections about common interests. The collections can be explored through a number of advanced Computer Vision and Machine Learning algorithms, able to capture the visual content of images in order to organize them in a semantic way. The interfaces of The Social Picture allow the users to create customized collections by exploiting semantic filters based on visual features, social network tags, geolocation, and other information related to the images.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130575564","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Philipp Blandfort, Tushar Karayil, Damian Borth, A. Dengel
{"title":"Introducing Concept And Syntax Transition Networks for Image Captioning","authors":"Philipp Blandfort, Tushar Karayil, Damian Borth, A. Dengel","doi":"10.1145/2911996.2930060","DOIUrl":"https://doi.org/10.1145/2911996.2930060","url":null,"abstract":"The area of image captioning i.e. the automatic generation of short textual descriptions of images has experienced much progress recently. However, image captioning approaches often only focus on describing the content of the image without any emotional or sentimental dimension which is common in human captions. This paper presents an approach for image captioning designed specifically to incorporate emotions and feelings into the caption generation process. The presented approach consists of a Deep Convolutional Neural Network (CNN) for detecting Adjective Noun Pairs in the image and a novel graphical network architecture called \"Concept And Syntax Transition (CAST)\" network for generating sentences from these detected concepts.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"14 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130892165","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Regional Subspace Projection Coding for Image Retrieval","authors":"Mingmin Zhen, Wenmin Wang, Ronggang Wang","doi":"10.1145/2911996.2912003","DOIUrl":"https://doi.org/10.1145/2911996.2912003","url":null,"abstract":"For image retrieval task, hamming embedding, being proved to be one of the state-of-the-art methods, has been prevalently utilised. The basic idea is to project local features into orthogonal space randomly, in which the binary signature is generated based on a single partition of feature space. However, the binary signature generation process is coarse and heuristic. On the one hand, the same projection is carried out for all visual word space without consideration of difference among subspaces. On the other hand, the projection matrix is generated randomly regardless of the distribution of feature data. Therefore, the performance of hamming embedding is limited and far from the optimal. In this paper, we firstly analyse the limitation of hamming em- bedding and compare different orthogonal projection methods. Then we propose a regional subspace projection coding method that is based on the distribution of local features assigned to each visual word. Finally, our experiments on two benchmark datasets demonstrate that our proposed method outperforms current state-of-the-art methods.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"5 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127858964","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Sequential Correspondence Hierarchical Dirichlet Processes for Video Data Analysis","authors":"Jianfei Xue, K. Eguchi","doi":"10.1145/2911996.2912041","DOIUrl":"https://doi.org/10.1145/2911996.2912041","url":null,"abstract":"Multimedia data mining based on topic models as an emerging technique has become a very popular research topic in recent years. In this paper, we propose a novel topic model named sequential correspondence hierarchical Dirichlet Processes (Seq-cHDP) to learn the hidden structure within video data. The Seq-cHDP model can be considered as an extended hierarchical Dirichlet processes (HDP) model containing two important features: one is the time-dependency mechanism that connects neighboring video frames on the basis of a time dependent Markovian assumption, and the other is the data correspondence mechanism that provides a solution for dealing with the multimodal data such as the mixture of visual words and speech words extracted from video files. We present a comprehensive evaluation for Seq-cHDP through experimentation and finally demonstrate that our model outperforms than other baseline models.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"79 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116095308","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Siripen Pongpaichet, Mengfan Tang, Laleh Jalali, R. Jain
{"title":"Using Photos as Micro-Reports of Events","authors":"Siripen Pongpaichet, Mengfan Tang, Laleh Jalali, R. Jain","doi":"10.1145/2911996.2912020","DOIUrl":"https://doi.org/10.1145/2911996.2912020","url":null,"abstract":"Photos serve dual role. Photos are important for capturing, saving, sharing, and reminiscing memories of events and people. Modern photos, however, are becoming more spontaneous, objective, compelling, and universal reports of a moment in an event also. In this paper our focus is on millions of photos being captured as informative reports and using them for emerging applications including situation recognition, trend analysis, and cultural dynamics. EventShop is an open source platform for situation recognition. Utilizing this platform and using a stream of photo reports from various sources as one of the data streams in this platform, we build a visual analytics system to understand the information that could be gleaned from such photo report streams. Our early experiments are based on the Yahoo Flickr Creative Commons 100 Million photos set released recently. We are also using other sources to import and understand the efficacy of these reports for various important applications.","PeriodicalId":201992,"journal":{"name":"Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval","volume":"100 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124910375","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}