MIGF-Net: Multimodal interaction-guided fusion network for image aesthetics assessment

Abstract

With the development of social media, people like to post images and comments to share their ideas, which provides rich visual and textural semantic information for image aesthetics assessment (IAA). However, most previous works either extracted the unimodal aesthetic features from image due to the difficulty of obtaining comments, or combined multimodal information together but ignoring the interactive relationship between image and comment, which limits the overall performance. To solve the above problem, we propose a Multimodal Interaction-Guided Fusion Network (MIGF-Net) for image aesthetics assessment based on both image and comment semantic information, which can not only solve the challenge of comment generating, but also provide the multimodal feature interactive information. Specifically, considering the coupling mechanism of the image theme, we construct a visual semantic fusion module to extract the visual semantic feature based on the visual attributes and the theme features. Then, a textural semantic feature extractor is designed to mine the semantic information hidden in comments, which not only addresses the issue of missing comments but also effectively complements the visual semantic features. Furthermore, we establish a Dual-Stream Interaction-Guided Fusion module to fuse the semantic features of images and comments, fully exploring the interactive relationship between images and comments in the human brain’s perception mechanism. Experimental results on two public image aesthetics evaluation datasets demonstrate that our model outperforms the current state-of-the-art methods. Our code will be released at https://github.com/wenzhipeng123/MIGF-Net.