A vision and language hierarchical alignment for multimodal aspect-based sentiment analysis

Abstract

In recent years, Multimodal Aspect-Based Sentiment Analysis (MABSA) has garnered attention from researchers. The MABSA technology can effectively perform Aspect Term Extraction (MATE) and Aspect Sentiment Classification (MASC) for Multimodal data. However, current MABSA work focuses on visual semantic information while neglecting the scene structure of images. Additionally, researchers using static alignment matrices cannot effectively capture complex vision features, such as spatial and action features among objects. In this paper, we propose a Vision and Language Hierarchical Alignment method (VLHA) for the MABSA task. The VLHA framework includes three modules: the multimodal structural alignment module, the multimodal semantic alignment module, and the cross-modal MABSA module. Firstly, we process the vision modality into a visual scene graph and image patches, and the text modality into a text dependency graph and word sequences. Secondly, we use the structural alignment module to achieve dynamic alignment learning between the visual scene graph and text dependency graph, and the semantic alignment module to achieve dynamic alignment learning between image patches and word sequences. Finally, we concatenate and fuse structural and semantic features in the cross-modal MABSA module. Additionally, VLHA designs a three-dimensional dynamic alignment matrix to guide the cross-attention for modal interaction learning. We conducted a series of experiments on two Twitter datasets, and the results show that the performance of the VLHA framework outperforms the baseline models. The structure of the visual modality facilitates the model in comprehensively understanding complex visual information.