A Multimodal Contrastive and Transfer Learning-Based Image Restoration Model for Multiple Adverse Weather Driving Scenes

Abstract

Adverse weather conditions like rain, fog, and snow significantly hinder perception in autonomous driving systems. This paper proposes a multimodal contrastive learning and transfer learning-based adaptive image restoration method for multiple adverse weather conditions. By integrating image and textual information, our method enhances robustness to diverse weather scenarios. Specifically, we first fine-tune a contrastive language-image pre-trained model to develop a multimodal image classifier capable of recognizing adverse weather conditions. Subsequently, an encoder-decoder-based restoration network is employed, where cross-attention layers incorporate textual conditional information, enabling the network to perceive weather variations. An adaptive restoration strategy is then applied to target specific noise characteristics associated with different weather conditions. Experiments on Rain Cityscapes, Foggy Cityscapes, and Snow Cityscapes show our model outperforms task-specific and All-in-One methods in visual and real-time performance, providing an efficient and robust solution for autonomous driving in complex environments.