Composed image retrieval by Multimodal Mixture-of-Expert Synergy

Composed image retrieval (CIR) is essential in security surveillance, e-commerce, and social media analysis. It provides precise information retrieval and intelligent analysis solutions for various industries. The majority of existing CIR models create a pseudo-word token from the reference image, which is subsequently incorporated into the corresponding caption for the image retrieval task. However, these pseudo-word-based prompting approaches are limited when the target image entails complex modifications to the reference image, such as object removal and attribute changes. To address the issue, we propose a Multimodal Mixture-of-Expert Synergy (MMES) model to achieve effective composed image retrieval. The MMES model initially utilizes multiple Mixture of Expert (MoE) modules through the mixture expert unit to process various types of multimodal input data. Subsequently, the outputs from these expert models are fused through the cross-modal integration module. Furthermore, the fused features generate implicit text embedding prompts, which are concatenated with the relative descriptions. Finally, retrieval is conducted using a text encoder and an image encoder. The Experiments demonstrate that the proposed method outperforms state-of-the-art CIR methods on the CIRR and Fashion-IQ datasets.