Visual-Aware Text as Query for Referring Video Object Segmentation

Current referring video object segmentation (R-VOS) approaches rely on directly identifying, locating, and segmenting referenced objects from text referring expressions in videos. However, there is inherent ambiguities in text referring expressions that can significantly negatively impact model performance. To address this challenge, a novel R-VOS method taking Visual-Aware Text as Query (VATaQ) is proposed, in which the referring expression is reconstructed with the guidance of visual feature, leading text feature to be highly relevant to the current video, thereby enhancing the clarity of the expressions. Furthermore, a CLIP-side Adapter Module (CAM), which leverages semantically enriched CLIP to enhance the visual feature with more semantic information, thus helping the model achieve a more comprehensive multi-modal representation. Experimental results show that the VATaQ shows outstanding performance on four video benchmark datasets, which outperforms the baseline network by 3.4% on the largest Ref-YouTube-VOS dataset.