Fine-grained knowledge progressive network with multivariate enhancements for text-based person retrieval

Objective:

Text-Based Person Retrieval (TBPR) aims to match target images using natural language descriptions, yet it faces significant challenges such as complex visual structures, diverse semantic expressions, and limited annotated data. These issues lead to intra-modal knowledge uncertainty and weak inter-modal correlations. Existing approaches predominantly focus on explicit alignment between heterogeneous modalities, often overlooking the latent associations within homogeneous knowledge. This study addresses these limitations to enhance both the performance and efficiency of TBPR.

Method:

This paper proposes a Multivariate Enhancement Fine-Grained Knowledge Progressive Network (ME-FKPN) to address the challenge of text-based person retrieval across modalities. ME-FKPN enhances the synergy between homogeneous and heterogeneous knowledge in a hierarchical manner, progressively establishing more accurate semantic alignments between images and texts. The framework comprises three key innovations: Standardized Knowledge Anchor (SKA) constructs a knowledge graph to standardize semantics; Mixed of LoRA Experts (MoLE) integrates defogged color and grayscale images to extract multi-level visual features; Multivariate Knowledge Progressive Optimization Strategy (MKPOS) achieves steady performance improvements through hierarchical augmentation and staged training.

Novelty:

The proposed ME-FKPN model introduces a novel solution to the TBPR task by integrating hierarchical knowledge enhancement and progressive optimization strategies. By deeply mining latent associations among homogeneous knowledge and fostering collaborative representation of heterogeneous knowledge, the approach effectively overcomes the limitations of existing techniques in handling complex cross-modal relationships and data sparsity.

Findings:

ME-FKPN outperforms all state-of-the-art TBPR models across three public datasets. Notably, on the challenging UFine6926 ultra-fine-grained dataset, our method achieves improvements of 16.49%, 9.79%, 6.23%, and 14.69% on R@1, R@5, R@10, and mAP metrics, respectively, compared to mainstream approaches.