Visual-Aware Text as Query for Referring Video Object Segmentation

IF 4.2 3区计算机科学 Q2 COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE

Image and Vision Computing Pub Date : 2025-06-17 DOI:10.1016/j.imavis.2025.105608

Qi Kuang, Ying Chen

引用次数: 0

Abstract

Current referring video object segmentation (R-VOS) approaches rely on directly identifying, locating, and segmenting referenced objects from text referring expressions in videos. However, there is inherent ambiguities in text referring expressions that can significantly negatively impact model performance. To address this challenge, a novel R-VOS method taking Visual-Aware Text as Query (VATaQ) is proposed, in which the referring expression is reconstructed with the guidance of visual feature, leading text feature to be highly relevant to the current video, thereby enhancing the clarity of the expressions. Furthermore, a CLIP-side Adapter Module (CAM), which leverages semantically enriched CLIP to enhance the visual feature with more semantic information, thus helping the model achieve a more comprehensive multi-modal representation. Experimental results show that the VATaQ shows outstanding performance on four video benchmark datasets, which outperforms the baseline network by 3.4% on the largest Ref-YouTube-VOS dataset.

查看原文本刊更多论文

视觉感知文本作为参考视频对象分割的查询

当前的引用视频对象分割（R-VOS）方法依赖于直接从视频中的文本引用表达式中识别、定位和分割引用对象。然而，在文本引用表达式中存在固有的歧义，这可能会对模型性能产生显著的负面影响。针对这一挑战，本文提出了一种基于视觉感知文本作为查询（VATaQ）的R-VOS方法，该方法在视觉特征的引导下重构参考表达，使文本特征与当前视频高度相关，从而提高表达的清晰度。此外，CLIP侧的适配器模块（CAM）利用语义丰富的CLIP，用更多的语义信息增强视觉特征，从而帮助模型实现更全面的多模态表示。实验结果表明，VATaQ在4个视频基准数据集上表现出了出色的性能，在最大的Ref-YouTube-VOS数据集上，VATaQ的性能比基线网络高出3.4%。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Image and Vision Computing 工程技术-工程：电子与电气

CiteScore

8.50

自引率

8.50%

发文量

143

审稿时长

7.8 months

期刊介绍： Image and Vision Computing has as a primary aim the provision of an effective medium of interchange for the results of high quality theoretical and applied research fundamental to all aspects of image interpretation and computer vision. The journal publishes work that proposes new image interpretation and computer vision methodology or addresses the application of such methods to real world scenes. It seeks to strengthen a deeper understanding in the discipline by encouraging the quantitative comparison and performance evaluation of the proposed methodology. The coverage includes: image interpretation, scene modelling, object recognition and tracking, shape analysis, monitoring and surveillance, active vision and robotic systems, SLAM, biologically-inspired computer vision, motion analysis, stereo vision, document image understanding, character and handwritten text recognition, face and gesture recognition, biometrics, vision-based human-computer interaction, human activity and behavior understanding, data fusion from multiple sensor inputs, image databases.