MetaFusion: Instance-Aware Meta-Prompting and Token Refinement for VLMs Generalization

Abstract

The rapid advancement of vision-language models (VLMs) has highlighted challenges in efficiently adapting these models to downstream tasks, where existing prompt-based fine-tuning (PFT) methods suffer from limited generalization to unseen classes due to homogeneous prompt representations and superficial prompt-feature interactions. To address these issues, we propose MetaFusion, a plug-and-play adapter module that jointly optimizes context-aware meta-prompt generation and feature refinement for enhanced visual-language alignment. Our method introduces a MetaNet to dynamically produce instance-specific meta-prompts from visual features, coupled with token aggregation guided with meta prompts (TAMPs) that semantically clusters and selects discriminative visual tokens through meta-prompt-guided grouping. In addition, a feature refinement module (FRM) fuses meta-prompts with visual features via cross-attention to amplify task-critical regions while suppressing background noise. Extensive experiments on 11 datasets demonstrate that MetaFusion significantly improves generalization capabilities when integrated with existing PFT frameworks, achieving state-of-the-art performance across zero-shot, few-shot, and domain-shift scenarios.