Dual global information guidance for deep contrastive multi-modal clustering

Abstract

Deep contrastive multi-modal clustering (MMC) leverages contrastive learning to capture complementary information across modalities. However, they face two challenges. First, there is a lack of eliminating irrelevant information for downstream tasks in the fused global information. Second, neglecting the guiding role of global information on local information can lead the model to fall into the situation of a locally optimal solution. To address these challenges, we propose a novel dual global information guidance for deep contrastive MMC (DGIG-CMMC) that leverages global information to explore the common and consistent information effectively. Global information contains comprehensive potential information, and DGIG-CMMC implements a global-to-local guidance to solve the problem of local insufficiency. The dual global information guidance includes global feature-guided feature-level (GloG-FeaLv) and global cluster assignments-guided label-level (GloG-LabLv). Specifically, GloG-FeaLv leverages global shared features to guide the private features of each modality, effectively eliminating irrelevant information for downstream tasks. GloG-LabLv employs global cluster assignments to guide and correct mistake partitions at the label-level, then ensuring the consistency and accuracy of clustering. Finally, all modules are optimized jointly in an end-to-end manner to enhance performance. Extensive experimental results confirm that the DGIG-CMMC method improves accuracy by 0.5% to 10.5% compared to state-of-the-art clustering approaches.