Visual residual aggregation network for visual-language prompt tuning

Prompt tuning leverages a series of learnable prompts to effectively guide pre-trained visual language models (VLMs) to adapt to various downstream tasks. VLMs encode deep features from both visual and textual branches and learn the joint embedding space of the two modalities by optimizing the contrast loss. However, existing prompt tuning methods face two critical challenges: (1) One challenge is the forgetting of generalized knowledge. As features propagate through the visual encoder, generalizable knowledge captured in shallow layers is gradually lost, ultimately impairing the generalization ability of the joint embedding space for new classes. (2) The other challenge is that models trained on the base class suffer from semantic bias. To address these issues, we propose Visual Residual Aggregation Network for Visual-Language Prompt Tuning (VraPT). VraPT comprises two sequentially connected components: a residual aggregation module and a semantic consistency module. Firstly, in order to solve the problem of generalized knowledge forgetting, the residual aggregation module enables adaptive fusion of generalized features, which effectively preserves generalized knowledge. It also reveals the importance of shallow features in enhancing the generalization capability of text prompts. The fused representation is then fed into the semantic consistency module which is used to address the problem of semantic bias. By minimizing the divergence from the true semantic distribution, this module enhances the semantic representations in the visual space as well as the semantic coherence of the learnable prompts. Our method enables the learned prompts to retain both discriminative semantic information and generalized knowledge. Extensive experiments show that our proposed VraPT is an effective prompt tuning method, especially in recognizing new classes with great improvement. On average, VraPT improves the accuracy on base classes by 1.06% and on new classes by 2.63% across 11 datasets, along with a 1.91% gain in the harmonic mean (H) metric.