One multimodal plugin enhancing all: CLIP-based pre-training framework enhancing multimodal item representations in recommendation systems

Abstract

With advances in multimodal pre-training, more efforts focus on integrating it into recommendation models. Current methods mainly focus on utilizing multimodal pre-training models to obtain multimodal representations of items and designing specific model architectures for downstream tasks. However, these methods often neglect the suitability of multimodal representations for recommendation systems since the pre-training is not conducted on recommendation datasets, making the directly obtained representations potentially suboptimal due to semantic biases from domain discrepancy and noise interference. Furthermore, collaborative information, a key element in recommendation systems, significantly impacts the effectiveness of recommendation models, but existing advanced multimodal pre-training models (e.g., CLIP) are unable to capture the collaborative information of items. To bridge the gap between multimodal pre-training models and recommendation systems, we propose a novel multimodal pre-training framework CLIP-based Pre-training MultiModal (CPMM) item representations model for recommendation. First, the representations of images, text, and IDs are mapped to a new low-dimensional contrastive representation space for alignment and semantic enhancement, ensuring the consistency and robustness of the multimodal contrastive representation (MCR). A contrastive learning approach is designed to regulate the inter-modal distances, mitigating the impact of noise on recommendation performance. Finally, modeling of the first-order similarities of the items is conducted, thereby integrating the collaborative information of the items into the multimodal contrastive representations. Extensive experiments on Amazon benchmark datasets (Beauty, Toys, Tools) validate CPMM’s effectiveness across three core recommendation tasks: sequential recommendation, collaborative filtering, and click-through rate prediction.