Vision-by-prompt: Context-aware dual prompts for composed video retrieval

Abstract

Composed video retrieval (CoVR) is a challenging task of retrieving relevant videos in a corpus by using a query that integrates both a relative change text and a reference video. Most existing CoVR models simply rely on the late-fusion strategy to combine visual and change text. Furthermore, various methods have been proposed to generate pseudo-word tokens from the reference video, which are then integrated into the relative change text for CoVR. However, these pseudo-word-based techniques exhibit limitations when the target video involves complex changes from the reference video, e.g., object removal. In this work, we propose a novel CoVR framework that learns context information via context-aware dual prompts for relative change text to achieve effective composed video retrieval. The dual prompts cater to two aspects: 1) Global descriptive prompts generated from the pretrained V-L models, e.g., BLIP-2, to get concise textual representations of the reference video. 2) Local target prompts to learn the target representations that the change text pays attention to. By connecting these prompts with relative change text, one can easily use existing text-to-video retrieval models to enhance CoVR performance. Our proposed framework can be flexibly used for both composed video retrieval (CoVR) and composed image retrieval (CoIR) tasks. Moreover, we take a pioneering approach by adopting the CoVR model to achieve zero-shot CoIR for remote sensing. Experiments on four datasets show that our approach achieves state-of-the-art performance in both CoVR and zero-shot CoIR tasks, with improvements of as high as around 3.5 % in terms of recall@K=1 score.