Mining fine-grained attributes for vision–semantics integration in few-shot learning

Abstract

Recent advancements in Few-Shot Learning (FSL) have been significantly driven by leveraging semantic descriptions to enhance feature discrimination and recognition performance. However, existing methods, such as SemFew, often rely on verbose or manually curated attributes and apply semantic guidance only to the support set, limiting their effectiveness in distinguishing fine-grained categories. Inspired by human visual perception, which emphasizes crucial features for accurate recognition, this study introduces concise, fine-grained semantic attributes to address these limitations. We propose a Visual Attribute Enhancement (VAE) mechanism that integrates enriched semantic information into visual features, enabling the model to highlight the most relevant visual attributes and better distinguish visually similar samples. This module enhances visual features by aligning them with semantic attribute embeddings through a cross-attention mechanism and optimizes this alignment using an attribute-based cross-entropy loss. Furthermore, to mitigate the performance degradation caused by methods that supply semantic information exclusively to the support set, we propose a semantic attribute reconstruction (SAR) module. This module predicts and integrates semantic features for query samples, ensuring balanced information distribution between the support and query sets. Specifically, SAR enhances query representations by aligning and reconstructing semantic and visual attributes through regression and optimal transport losses to ensure semantic–visual consistency. Experiments on five benchmark datasets, including both general datasets and more challenging fine-grained Few-Shot datasets consistently demonstrate that our proposed method outperforms state-of-the-art methods in both 5-way 1-shot and 5-way 5-shot settings.