CoCNet: A Chain-of-Clues framework for zero-shot referring expression comprehension

Abstract

Zero-shot learning enables the reference expression comprehension (REC) model to adapt to a wide range of visual domains without training. However, the ambiguity of linguistic expression leads to the lack of a clear subject. Moreover, existing methods have not fully utilized the visual context and spatial information, resulting in low accuracy and robustness in complex scenes. To address these problems, we propose a Chain-of-Clues framework (CoCNet) to exploit multiple clues for zero-shot REC task to solve the inference confusion step by step. First, the subject clue module employs the strong ability of large language models (LLMs) to reason about the category in expression, which enhances the clarity of linguistic expression. In the attribute clue module, we propose the dual-track scoring which highlights the proposal by blurring its surroundings and enhances contextual sensitivity by blurring the proposal. Additionally, the spatial clue module utilizes a series of Gaussian-based soft heuristic rules to model the location words and the spatial relationship of the image. Experimental results show that CoCNet exhibits strong generalization capabilities in complex scenes. It significantly outperforms previous state-of-the-art zero-shot methods on RefCOCO, RefCOCO+, RefCOCOg, Flickr-Split-0 and Flickr-Split-1. Our code is released at https://github.com/CoCNetHub/CoCNet-main.