{"title":"Learning Class Prototypes Via Anisotropic Combination of Aligned Modalities for Few-Shot Learning","authors":"Jieya Lian, Haojie Wang, Shengwu Xiong","doi":"10.1109/icme46284.2020.9102883","DOIUrl":"https://doi.org/10.1109/icme46284.2020.9102883","url":null,"abstract":"Prototypical networks have shown their simplicity and effectiveness in few-shot learning. Recently, some works leverage cross-modal information to enhance the class prototype in few-shot learning. However, they seldom make use of structured information from textual space to optimize class prototype representation. And they either don’t align textual modality to visual modality or align them too rigidly. We argue that proper alignment method is important to improve the performance of cross-modal methods, as query data only has visual information in the few-shot learning tasks. In this paper, we propose a cross-modal alignment method to optimize class prototypes with structured information from textual space. We further introduce an anisotropic combination method to enhance class prototypes with information from two modalities. Experiments show that the cross-modal alignment method and anisotropic combination method achieve state-of-the-art results on the miniImageNet and tieredImageNet benchmark in one-shot regime.","PeriodicalId":396992,"journal":{"name":"2020 IEEE International Conference on Multimedia and Expo (ICME)","volume":"85 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115616284","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Compressed Video Sensing Network Based On Alignment Prediction And Residual Reconstruction","authors":"Xinpeng Ling, Chunling Yang, Hanqi Pei","doi":"10.1109/ICME46284.2020.9102723","DOIUrl":"https://doi.org/10.1109/ICME46284.2020.9102723","url":null,"abstract":"Image compressed sensing networks based on deep learning have achieved promising performance recently, which provides a new idea for Compressed Video Sensing (CVS). For CVS, the reconstruction performance can be improved by utilizing the correlation between adjacent frames. Following the traditional ‘prediction-residual reconstruction’ framework, we propose a novel learnable network for CVS based on alignment prediction and residual reconstruction named PRCVSNet. In PRCVSNet, Temporal Deformable Alignment Network (TDAN) is introduced to predict the current frame from reference frames for the first attempt. Furthermore, we design a novel residual reconstruction mechanism which can be applied in both initial and enhancement reconstruction stages to improve the reconstruction performance of video with the measurements. At last, the PRCVSNet introduces a fusion network taking advantage of information of multiple reference frames. Experimental results demonstrate the effectiveness of our network against the state-of-art methods with improvement of 1-5dB in PSNR.","PeriodicalId":396992,"journal":{"name":"2020 IEEE International Conference on Multimedia and Expo (ICME)","volume":"31 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117192269","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Xiaoyu Chai, Jun Chen, Chao Liang, Dongshu Xu, Chia-Wen Lin
{"title":"Expression-Aware Face Reconstruction Via A Dual-Stream Network","authors":"Xiaoyu Chai, Jun Chen, Chao Liang, Dongshu Xu, Chia-Wen Lin","doi":"10.1109/icme46284.2020.9102811","DOIUrl":"https://doi.org/10.1109/icme46284.2020.9102811","url":null,"abstract":"Recently, 3D face reconstruction from a single image has achieved promising results by adopting the 3D Morphable Model (3DMM). However, as face images in-the-wild have various expressions, it is difficult for 3DMM to handle diverse facial expressions with a large range of variations, due to the limited expressive ability of its linear model, thereby resulting in distortion and ambiguity on facial local regions. To tackle this issue, we present a novel dual-stream network to deal with expression variations. Specifically, in the geometry stream, we propose novel Attribute Spatial Maps to record the spatial information of facial identity and expression attributes in the 2D image space separately. This avoids the interaction between the two attributes, thus keeping the identity information and further improving the ability to cope with expression changes. In the texture stream, we utilize the 3DMM albedo map to a style transfer based method for synthesizing facial appearance, which results in expression-irrelevant as well as realistic face textures. Both quantitative and qualitative evaluations on public datasets demonstrate the ability of our approach to achieve comparable results in face reconstructions under expression variations.","PeriodicalId":396992,"journal":{"name":"2020 IEEE International Conference on Multimedia and Expo (ICME)","volume":"16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117222427","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Joint Facial Action Unit Intensity Prediction And Region Localisation","authors":"Yachun Fan, Jie Shen, Housen Cheng, Feng Tian","doi":"10.1109/icme46284.2020.9102833","DOIUrl":"https://doi.org/10.1109/icme46284.2020.9102833","url":null,"abstract":"Facial Action Unit (AU) intensity prediction is essential to facial expression analysis and emotion recognition, and thus has attracted much attention from the community. In comparison, AU localization, albeit being important to emotion visualization and tracking, was relatively unexplored. In addition, as most existing AU intensity prediction methods take a cropped face image as input, their actual speed at run-time is often penalized by the pre-processing steps such as face detection and alignment. At the same time, their performance (in terms of inference speed), also does not scale well to multi-face images. To alleviate these problems, we propose a joint AU intensity prediction and localization method that works directly on the whole input image, thus eliminating the need of any pre-processing step and achieving the same inference speed regardless of the number of faces in the image. Based on the observation that different relevancy exists between AU intensities categories, a flexible cost function is proposed. At inference time, we introduce a non-maximum intensity suppression model to refine the prediction result. In order to leverage existing datasets without AU region groundtruth, we also propose an automatic AU region labeling method. Experiments on two benchmark databases, DISFA and FERA2015, show that the proposed approach outperforms the state-of-the-art methods on three metrics, ICC, MAE and F1 for the AU intensity prediction task.","PeriodicalId":396992,"journal":{"name":"2020 IEEE International Conference on Multimedia and Expo (ICME)","volume":"82 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123757409","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Style-Conditioned Music Generation","authors":"Yu-Quan Lim, Chee Seng Chan, F. Loo","doi":"10.1109/ICME46284.2020.9102870","DOIUrl":"https://doi.org/10.1109/ICME46284.2020.9102870","url":null,"abstract":"Recent works have shown success in generating music using a Variational Autoencoder (VAE). However, we found out that the style of the generated music is usually governed or limited by the training dataset. In this work, we proposed a new formulation to the VAE that allows users to condition on the style of the generated music. Technically, our VAE consists of two latent spaces- content and style space to encode the content and style of a song separately. Each style is represented by a continuous style embedding, unlike previous works which mostly used discrete or one-hot style labels. We trained our model on public datasets that made up of Bach chorales and western folk tunes. Empirically, as well as from music theory point of view, we show that our proposed model can generate better music samples of each style than a baseline model. The source code and generated samples are available at https: //git;hub. com/daQuincy /DeepMu sicvSt; yle","PeriodicalId":396992,"journal":{"name":"2020 IEEE International Conference on Multimedia and Expo (ICME)","volume":"61 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115564411","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Shaofeng Zhang, Zheng Wang, Xing Xu, Xiang Guan, Yang Yang
{"title":"Fooled by Imagination: Adversarial Attack to Image Captioning Via Perturbation in Complex Domain","authors":"Shaofeng Zhang, Zheng Wang, Xing Xu, Xiang Guan, Yang Yang","doi":"10.1109/icme46284.2020.9102842","DOIUrl":"https://doi.org/10.1109/icme46284.2020.9102842","url":null,"abstract":"Adversarial attacks are very successful on image classification, but there are few researches on vision-language systems, such as image captioning. In this paper, we study the robustness of a CNN+RNN based image captioning system being subjected to adversarial noises in complex domain. In particular, we propose Fooled-by-Imagination, a novel algorithm for crafting adversarial examples with semantic embedding of targeted caption as perturbation in complex domain. The proposed algorithm explores the great merit of complex values in introducing imaginary part for modeling adversarial perturbation, and maintains the similarity of the image in real part. Our approach provides two evaluation approaches, which check whether neural image captioning systems can be fooled to output some randomly chosen captions or keywords. Besides, our method has good transferability under black-box setting. At last, our extensive experiments show that our algorithm can successfully craft visually-similar adversarial examples with randomly targeted captions or keywords at a higher success rate.","PeriodicalId":396992,"journal":{"name":"2020 IEEE International Conference on Multimedia and Expo (ICME)","volume":"26 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116841177","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A Study Of Parking-Slot Detection With The Aid Of Pixel-Level Domain Adaptation","authors":"Juntao Chen, Lin Zhang, Ying Shen, Yong Ma, Shengjie Zhao, Yicong Zhou","doi":"10.1109/icme46284.2020.9102928","DOIUrl":"https://doi.org/10.1109/icme46284.2020.9102928","url":null,"abstract":"The self-parking system is an important component of self-driving vehicles. Such a system needs to detect and locate the parking-slots from surround-view images, and then guide the vehicle to the designated parking-slot. In the real world, the appearances and environmental conditions of parking-slots can be rich and varied. Thus, to train the parking-slot detection model, it is necessary to collect and label a huge quantity of surround-view images covering as many real cases as possible. Such a process is cumbersome and costly, and will be repeated whenever encountering an unseen parking condition that is quite different from the ones covered by existing training set. To this end, in this paper we propose an extensible pipeline, namely FakePS, to assist parking-slot detection model training by making use of synthetic data. Specifically, with FakePS, we can first build various simulated parking scenes and collect labeled surround-view images automatically. Besides, we resort to pixel-level domain adaptation strategies to enhance the realism of the synthetic images using unlabeled real images while preserving their label information. The efficacy of FakePS has been corroborated by experimental results.","PeriodicalId":396992,"journal":{"name":"2020 IEEE International Conference on Multimedia and Expo (ICME)","volume":"23 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129415991","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Yongzhi Li, Lu Chi, Guiyu Tian, Yadong Mu, Shen Ge, Zhi Qiao, Xian Wu, Wei Fan
{"title":"Spectrally-Enforced Global Receptive Field For Contextual Medical Image Segmentation And Classification","authors":"Yongzhi Li, Lu Chi, Guiyu Tian, Yadong Mu, Shen Ge, Zhi Qiao, Xian Wu, Wei Fan","doi":"10.1109/ICME46284.2020.9102949","DOIUrl":"https://doi.org/10.1109/ICME46284.2020.9102949","url":null,"abstract":"Deep convolutional neural networks (CNNs) have recalibrated the state-of-the-art for a plethora of applications in medical image analyzing such as segmentation and classification. Large receptive field is crucial for modeling long-range spatial dependency in medical images. In this paper, we propose a novel architectural network design for accomplishing a full-image global receptive field, which we call spectral residual block (SRB). Specifically, we propose to utilize a unitary transform that essentially conducts a local-to-global transform. All elements are mapped to spectral domain and thus globally depend on each other. A variety of global operators are carefully devised and efficiently enforce a full-image receptive field, including spectral ReLU for frequency-sensitive filtering and spectral convolutions. The output in spectral domain is eventually converted back global-to-local via a reverse unitary transform. The proposed framework is generic and flexible, and could be applied to various network structures and tasks. Comprehensive evaluations on skin lesion segmentation and Chest X-Ray classification show that our method achieves the state-of-the-art performance, demonstrating both effectiveness and efficiency.","PeriodicalId":396992,"journal":{"name":"2020 IEEE International Conference on Multimedia and Expo (ICME)","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130383735","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Region Adaptive Two-Shot Network For Single Image Dehazing","authors":"Hui Li, Q. Wu, K. Ngan, Hongliang Li, Fanman Meng","doi":"10.1109/ICME46284.2020.9102944","DOIUrl":"https://doi.org/10.1109/ICME46284.2020.9102944","url":null,"abstract":"Existing single image dehazing methods typically adopt a one-shot strategy by indiscriminately applying the same filters to all local regions, which easily cause under-/over-dehazing across different regions by ignoring the inhomogeneity and asymmetry of illumination and detail distortions. In this paper, we propose a region adaptive two-shot network (RATNet) to address this issue. In the first shot, a lightweight subnetwork is utilized to conduct the regular global filtering, which could remove parts of haze but also distort some image details. In the second shot, a two-branch subnetwork is developed to restore the illumination and details of the initially renovated image respectively. The final dehazed image is obtained by fusing the outputs of the previous two branches, whose region-variant weights are adaptively learned by minimizing the difference between the haze-free image and our fused result. Experiments on four dehazing benchmark datasets show that our RATNet significantly outperforms many state-of-the-art dehazing approaches.","PeriodicalId":396992,"journal":{"name":"2020 IEEE International Conference on Multimedia and Expo (ICME)","volume":"49 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130584672","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Video Anomaly Detection Via Predictive Autoencoder With Gradient-Based Attention","authors":"Yuandu Lai, R. Liu, Yahong Han","doi":"10.1109/ICME46284.2020.9102894","DOIUrl":"https://doi.org/10.1109/ICME46284.2020.9102894","url":null,"abstract":"Video anomaly detection is a challenging problem due to the ambiguity and diversity of anomalies in different scenes. In this paper, we present a novel framework to detect abnormal in surveillance videos. Inspired by the common deep reconstruction methods and deep prediction ones, we propose a new two-branch predictive autoencoder, including a reconstruction decoder and a prediction decoder, in which the prediction decoder is used to generate future frame and carry out anomaly detection by comparing the difference between predicted future frame and its ground truth. And the reconstruction decoder reconstructs the current frame, which can constrains the encoder to learn video representations better. Moreover, reconstruction decoder provides a gradient-based attention, which significantly helps the prediction decoder to generate higher quality future frame. Our method unifies reconstruction and prediction methods in an end-to-end framework, and it obtains impressive results with better predicted future frame on some publicly available datasets including CUHK Avenue and UCSD Pedestrian.","PeriodicalId":396992,"journal":{"name":"2020 IEEE International Conference on Multimedia and Expo (ICME)","volume":"26 16","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"120860993","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}