Instance-aware context with mutually guided vision-language attention for referring image segmentation

Abstract

Referring image segmentation, which integrates both visual and linguistic modalities, represents a forefront challenge in cross-modal visual research. Traditional approaches generally fuse linguistic features with visual data to generate multi-modal representations for mask decoding. However, these methods often mistakenly segment visually prominent entities rather than the specific region indicated by the referring expression, as the visual context tends to overshadow the multi-modal features. To address this, we introduce IMNet, a novel referring image segmentation framework that harnesses the Contrastive Language-Image Pre-training (CLIP) model and incorporates a mutually guided vision-language attention mechanism to enhance accuracy in identifying the referring mask. Specifically, our mutually guided vision-language attention mechanism consists of language-guided attention and vision-guided attention, which model bi-directional relationships between vision and linguistic features. Additionally, to accurately segment instances based on referring expressions, we develop an instance-aware context module within the decoder that focuses on learning instance-specific features. This module connects instance prototypes with corresponding features, using linearly weighted prototypes for final prediction. We evaluate the proposed method on three publicly available datasets, i.e., RefCOCO, RefCOCO+, and G-Ref. Comparisons with previous methods demonstrates that our approach achieves competitive performance.