Dynamic T-distributed stochastic neighbor graph convolutional networks for multi-modal contrastive fusion

Abstract

As the continuous advancement of data acquisition technologies progresses, multi-modal data have emerged as a prominent focus in various domains. This paper aims to tackle critical challenges in the multi-modal fusion process, specifically in representation learning, modal consistency invariance learning, and model diversity complementarity learning, by employing graph convolutional networks and contrastive learning methods. Current GCN-based methods generally depend on predefined graphs for representation learning, limiting their capacity to capture local and global information effectively. Furthermore, some current models do not adequately compare the representations of consistency and diversity across different modalities during the fusion procedure. To address the identified challenges, we propose a novel T-distributed Stochastic Neighbor Contrastive Graph Convolutional Network (TSNGCN). It consists of the adaptive static graph learning module, the multi-modal representation learning module, and the multi-modal contrastive fusion module. The adaptive static graph learning module constructs graphs without relying on any predefined distance metrics, which creates a pairwise graph adaptively to preserve the local structure of general data. Moreover, a loss function based on T-distributed stochastic neighbor embedding is designed to learn the transformation between the embeddings and the original data, thus facilitating the exploration of more discriminative information within the learned subspace. In addition, the proposed multi-modal contrastive fusion module effectively maximizes the similarity of the same samples across different modalities while ensuring the distinction of dissimilar samples, thereby enhancing the model’s consistency objective. Extensive experiments conducted on several multi-modal benchmark datasets demonstrate the superiority and effectiveness of TSNGCN compared to existing methods.