Instruction-guided fusion of multi-layer visual features in Large Vision-Language Models

Abstract

Large Vision-Language Models (LVLMs) have achieved remarkable success in a wide range of multimodal tasks by integrating pre-trained vision encoders with large language models. However, current LVLMs primarily rely on visual features extracted from the final layers of the vision encoder, overlooking the complementary information available in shallower layers. While recent works have explored the fusion of multi-layer visual features in LVLMs, they typically adopt task-agnostic fusion strategies and do not examine the dependencies of these features on different tasks. To address these gaps, we systematically investigate the individual contributions of hierarchical visual features in the context of LVLMs. Our findings reveal that the visual features from different layers exhibit complementary strengths, with their effectiveness varying across different tasks. Motivated by these insights, we introduce an instruction-guided vision aggregator that dynamically fuses multi-layer visual features based on task-specific instructions. Extensive evaluations demonstrate the superior performance of our method. We hope this work will provide valuable insights into the adaptive use of hierarchical visual features in LVLMs. Our code is available at: https://github.com/YiZheng-zy/IGVA.