CLEAR: Cross-Transformers With Pre-Trained Language Model for Person Attribute Recognition and Retrieval

Abstract

Person attribute recognition and attribute-based person retrieval are two core human-centric tasks. In the recognition task, the challenge lies in identifying attributes based on a person’s appearance, while the retrieval task involves searching for matching persons using attribute-based queries. In this paper, we present CLEAR, a unified network designed to address both tasks. We leverage our C${}^2$T-Net, a strong Cross-Transformers backbone that achieved state-of-the-art performance in the person attribute recognition task during the UPAR Challenge 2024, to extract visual embeddings. We then adapt it for the attribute-based person retrieval task.To extend its capabilities for the attribute-based person retrieval task, we construct pseudo-textual descriptions for attribute queries, leverage a pretrained language model to generate language-rich feature embeddings, and introduce an effective training strategy, which involves finetuning only a few additional parameters in the form of adapters to produce visual and query embeddings within the retrieval space. As the visual embeddings extracted by C${}^2$T-Net are highly discriminative, they align well with the proposed query embeddings during the finetuning process, facilitating improved retrieval performance.The unified CLEAR, model is evaluated on five benchmarks: PETA, PA100K, Market-1501, RAPv2, and UPAR2024, achieving state-of-the-art or competitive results for both tasks. Notably, it ranks as the top performer on the large-scale UPAR2024 dataset, specifically designed to test domain generalizability in real-world scenarios where test samples differ from training samples.