Learning relationship-guided vision-language transformer for facial attribute recognition

Facial Attribute Recognition (FAR) remains a challenging task due to the poor quality of visual images. Existing FAR methods generally adopt the Convolutional Neural Network (CNN) to learn deep features of facial images, and attribute relationships are either obtained by a fixed clustering algorithm or manually grouped. In this paper, we propose a novel Relationship-Guided Vision-Language Transformer, termed RVLT, for FAR. The RVLT method can automatically learn attribute relationships from the linguistic modality, which is used to guide image feature extraction. Specifically, each vision-language Transformer encoder adopts an Image-Text Cross-Attention (ITCA), which is composed of an Image-to-Text Adjustment Attention (ITAA) and a Text-to-Image Guidance Attention (TIGA). The ITAA is used to adjust the text tokens from the linguistic modality to adapt to the visual information, and the TIGA employs the adjusted text tokens as the prior knowledge to guide the distribution of the image embeddings from the visual modality. Moreover, we employ a Token Selection Mechanism (TSM) to reduce the interference factors in the image background, so that the model can pay more attention to the regions related to the face. In addition, an Image-Text Alignment (ITA) loss is used to further align the tokens of the visual and linguistic modalities, and a Text-Aware Classification (TAC) loss is leveraged to ensure the correctness of the attribute relationships learned from the raw text as much as possible. Different from directly using the image embeddings for feature extraction, our method can leverage the attribute relationships automatically learned from the raw text to effectively highlight image features relevant to facial attributes. Experimental results demonstrate that the proposed RVLT achieves 87.34% and 91.80% accuracy on LFWA and CelebA, respectively. In the case of limited labeled data, RVLT outperforms the second-best FAR method by 1.76% and 0.25% accuracy under only 5% LFWA and 0.5% CelebA training data, respectively.