Multimodal Aesthetic Analysis Assisted by Styles through a Multimodal co-Transformer Model

Abstract

Many real-world applications could profit from the ability of image aesthetic analysis. A simultaneous understanding of both the visual content of images and the textual content of user comments and style attributes appears to be more vivid and adequate than single-modality and single-dimension information to help people learning to identify beauty or not. In this paper, we propose a multimodal co-transformer model to learn a joint representation of multimodal contents based on the co-attention mechanism, and then we conduct multi-dimension aesthetic analysis assisted by style attributes. Towards this goal, we propose a stacked multimodal co-transformer module encoding the feature under interactive guidance, and then we utilize a multi-task learning strategy for predicting multiple aesthetic dimensions. Experimental results indicate that the proposed model achieves state-of-the-art performance on the AVA datasets benchmark.