SDDA: A progressive self-distillation with decoupled alignment for multimodal image–text classification

Abstract

Multimodal image–text classification endeavors to deduce the correct category based on the information encapsulated in image–text pairs. Despite the commendable performance achieved by current image–text methodologies, the intrinsic multimodal heterogeneity persists as a challenge, with the contributions from diverse modalities exhibiting considerable variance. In this study, we address this issue by introducing a novel decoupled multimodal Self-Distillation (SDDA) approach, aimed at facilitating fine-grained alignment of shared and private features of image–text features in a low-dimensional space, thereby reducing information redundancy. Specifically, each modality representation is decoupled in an autoregressive manner into two segments within a modality-irrelevant/exclusive space. SDDA imparts additional knowledge transfer to each decoupled segment via self-distillation, while also offering flexible, richer multimodal knowledge supervision for unimodal features. Multimodal classification experiments conducted on two publicly available benchmark datasets verified the efficacy of the algorithm, demonstrating that SDDA surpasses the state-of-the-art baselines.