Object-Level and Scene-Level Feature Aggregation with CLIP for scene recognition

Abstract

Scene recognition is a fundamental task in computer vision, pivotal for applications like visual navigation and robotics. However, traditional methods struggle to effectively capture and aggregate scene-related features due to the inherent complexity and diversity of scenes, often leading to sub-optimal performance. To address this limitation, we propose a novel method, named OSFA (Object-level and Scene-level Feature Aggregation), that leverages CLIP’s multimodal strengths to enhance scene feature representation through a two-stage aggregation strategy: Object-Level Feature Aggregation (OLFA) and Scene-Level Feature Aggregation (SLFA). In OLFA, we first generate an initial scene feature by integrating the average-pooled feature map of the base visual encoder and the CLIP visual feature. The initial scene feature is then used as a query in object-level cross-attention to extract object-level details most relevant to the scene from the feature map, thereby enhancing the representation. In SLFA, we first use CLIP’s textual encoder to provide category-level textual features for the scene, guiding the aggregation of corresponding visual features from the feature map. OLFA’s enhanced scene feature then queries these category-aware features using scene-level cross-attention to further capture scene-level information and obtain the final scene representation. To strengthen training, we employ a multi-loss strategy inspired by contrastive learning, improving feature robustness and discriminative ability. We evaluate OSFA on three challenging datasets (i.e. Places365, MIT67, and SUN397), achieving substantial improvements in classification accuracy. These results highlight the effectiveness of our method in enhancing scene feature representation through CLIP-guided aggregation. This advancement significantly improves scene recognition performance. Our code is public at https://github.com/WangqunQAQ/OSFA.