Multi-SEA: Multi-stage Semantic Enhancement and Aggregation for image–text retrieval

Abstract

Image–text retrieval aims to find a general embedding space to semantically align cross-modal tokens. Existing studies struggle to adequately integrate information cross different modality encoders and usually neglect implicit semantic information mining, making it difficult to accurately understand and represent cross-modal information. To resolve the problems mentioned above, we propose a Multi-stage Semantic Enhancement and Aggregation framework (Multi-SEA) with novel networks and training schemes, which can more comprehensively integrate global and local information within both intra-modal and inter-modal features. Multi-SEA first designs a fusion module with agent attention and gating mechanism. It helps the model focus on crucial information. Multi-SEA then introduces a three-stage scheme to enhance uni-modal information and aggregates fine-grained cross-modal information by involving the fusion module in different stages. Eventually, Multi-SEA utilizes a negative sample queue and hierarchical scheme to facilitate robust contrastive learning and promote expressive capabilities from implicit information. Experimental results demonstrate that Multi-SEA significantly outperforms the state-of-the-art schemes, achieving notable improvements in image-to-text and text-to-image retrieval tasks on the Flickr30k, MSCOCO (1K), and MSCOCO (5K) datasets, with Recall@sum increased by 13.3, 2.8, and 4.7, respectively.