Multi-Layer Cross-Modal Prompt Fusion for No-Reference Image Quality Assessment

Abstract

No-Reference Image Quality Assessment (NR-IQA) predicts image quality without reference images and exhibits high consistency with human visual perception. Multi-modal approaches based on vision-language (VL) models, like CLIP, have demonstrated remarkable generalization capabilities in NR-IQA tasks. While prompt learning has improved CLIP’s adaptation to downstream tasks, existing methods often lack synergy between textual and visual prompts, limiting their ability to capture complex cross-modal semantics. In response to this limitation, this paper proposes an innovative framework named MCPF-IQA with multi-layer cross-modal prompt fusion to further enhance the performance of CLIP model on NR-IQA tasks. Specifically, we introduce multi-layer prompt learning in both the text and visual branches of CLIP to improve the model’s comprehension of visual features and image quality. Additionally, we design a novel cross-modal prompt fusion module that deeply integrates text and visual prompts to enhance the accuracy of image quality assessment. We also develop five auxiliary quality-related category labels to describe image quality more precisely. Experimental results demonstrate MCPF-IQA model delivers exceptional performance on natural image datasets, with SRCC of 0.988 on the LIVE dataset (1.8% higher than the second-best method) and 0.913 on the LIVEC dataset (1.0% superior to the second-best method). Furthermore, it also exhibits strong performance on AI-generated image datasets. Ablation study results demonstrate the effectiveness and advantages of our method.