Adaptive integration of textual context and visual embeddings for underrepresented vision classification

Abstract

The advancement of deep learning has significantly improved image classification performance; however, handling long-tail distributions remains challenging due to the limited data available for rare classes. Existing approaches predominantly focus on visual features, often neglecting the valuable contextual information provided by textual data, which can be especially beneficial for classes with sparse visual examples. In this work, we introduce a novel method addressing this limitation by integrating textual data generated by advanced language models with visual inputs through our newly proposed Adaptive Integration Block for Vision-Text Synergy (AIB-VTS). Specifically designed for Vision Transformer architectures, AIB-VTS adaptively balances visual and textual information during inference, effectively utilizing textual descriptions generated from large language models. Extensive experiments on benchmark datasets demonstrate substantial performance improvements across all class groups, particularly in underrepresented (tail) classes. These results confirm the effectiveness of our approach in leveraging textual context to mitigate data scarcity issues and enhance model robustness.