Group-guided prompt learning for vision-language models

Abstract

Prompt learning has become one of the mainstream approaches for enabling Vision-Language Models (VLMs) to effectively adapt to downstream tasks. Recent approaches enhanced the generalization of models by integrating prior knowledge from large language models (LLMs). However, these approaches overlook the potential value of group knowledge derived from semantic correlations across different classes, which may limit the performance of the model in the face of complex downstream tasks. To overcome this challenge, we propose Group-guided Prompt Learning (GGPL), which integrates group knowledge into the original text prompts through LLMs. Specifically, GGPL uses LLMs to group all classes and integrates the group knowledge into the original text prompts to construct the final text prompts. Furthermore, we introduce a novel Group Knowledge Alignment (GKA) module, which aligns the learnable prompt features with the pre-trained features that contain group knowledge, preventing the learnable prompt features from feature shift during the training process and thus reducing overfitting. Experimental results across 11 public datasets demonstrate that the proposed GGPL method achieves significant improvement on various prompt learning approaches, while numerous ablation experiments also demonstrate the effectiveness of the each component of our GGPL method.