Dynamic sparse and weight allocation-based text-driven person retrieval

Abstract

Text-to-image person retrieval aims to find the most matching personimages in a large-scale persondataset through textual descriptions. However, most of the existing methods have the following problems: (1) There are still some inaccurate matching pairs in the retrieval system, and the errors of these matching pairs negatively affect the performance of the whole retrieval system. (2) In the whole training process of the model, the whole text is used directly, but there are still non-important parts of the text that are not important for recognizing the images, and how to process the text effectively is still a hot topic in current research. These critical issues significantly degrade the retrieval performance. To this end, we propose a new alignment optimization framework for text-based person retrieval. Precisely, our framework consists of three key components: (1) progressive enhancement for a multimodal integration, which not only simulates coarse-grained alignment through mathematical modeling, but also appropriately combines coarse-grained and fine-grained alignment through progressive learning; (2) global bidirectional match filtering, which utilizes subjective logic to effectively mitigate the interference of incorrectly matched pairs of image text, and at the same time utilizes a bidirectional KL match filtering algorithm so as to select the matching pairs with high degree of image text matching for training; (3) fine-grained dynamic sparse mask modeling, which uses mask language modeling and constructs a dynamic spatial sparsification module, which not only applies more expressive modules to important positions but also mines the relationship between image text pairs at a fine-grained level, thus improving retrieval performance. Extensive experiments show that the method achieves state-of-the-art results on three benchmark datasets and performs well on domain generalization tasks.