VCGPrompt: Visual Concept Graph-Aware Prompt Learning for Vision-Language Models

Prompt learning enables efficient fine-tuning of visual-language models (VLMs) like CLIP, demonstrating strong transferability across varied downstream tasks. However, adapting VLMs to open-vocabulary tasks is challenging due to the requirement to recognize diverse unseen data, which can cause overfitting and hinder generalization. To address this, we propose Visual Concept Graph-Aware Prompt Learning (VCGPrompt), which constructs visual concept graphs and uses fine-grained text prompts to enrich the general world knowledge of the model. Additionally, we introduce the Visual Concept Graph Aggregation Module (VCGAM) to prioritize the most distinctive visual concepts of each category and guide the learning of relevant visual features, which enhances the capability to perceive the open world. Our method achieves consistent improvements across three diverse generalization settings, including base-to-new, cross-dataset, and domain generalization, with performance gains of up to 0.95%. These results demonstrate the robustness and broad applicability of our approach under various scenarios. Detailed ablation studies and analyses validate the necessity of fine-grained prompts in the open-vocabulary setting.