InterCLIP: Adapting CLIP To Interactive Image Retrieval with Triplet Similarity

Abstract

Interactive image retrieval is such task setting where a multi-modal query (reference image, feedback text) is provided, and the goal is to retrieve a target image which satisfies the changes described in feedback text based on the reference image. It offers a great promise for better user experience in a variety of fields such as e-commerce where the user can address their need with natural language and find the desired item iteratively. With the rising of Vision-Language Pre-trained(VLP) models, it has become a de facto to transfer rich knowledge learned from large-scale real-world data to downstream tasks. In this work, we propose a novel method called InterCLIP, which adapt the matching oriented VLP model CLIP, to the task. To further harness the power of CLIP, we propose to view the task as a combination of text-image retrieval and standard image search. Specifically we calculate candidate images’ similarity score with similarity within the triplet. This method allows fine-grained modelling which takes account of the relevance between three pairs within the triplet, and extensive experiments show our method achieves state-of-the-art results on the FashionIQ dataset.