MALM-CLIP: A generative multi-agent framework for multimodal fusion in few-shot industrial anomaly detection

Abstract

The Contrastive Language-Image Pre-training (CLIP) model has significantly improved few-shot industrial anomaly detection. However, existing approaches often rely on manually crafted visual description texts, which lack robustness and generalizability in real-world production settings. This limitation is evident as these methods struggle to adapt to new or evolving anomalies, where original prompts fail to generalize beyond their initial design. This paper proposes a novel method, Multi-agent Language Models with CLIP (MALM-CLIP), which integrates the generative capabilities of large language models (LLMs) with CLIP within a multi-agent framework. In this system, specialized agents handle different subtasks such as prompt generation and model evaluation, enabling automated and context-aware multimodal information fusion. By eliminating manual prompt engineering, MALM-CLIP enhances both the accuracy and efficiency of anomaly detection. Experimental results on standard datasets such as MVTec and VisA demonstrate that our approach outperforms existing methods in detecting image-level anomalies with minimal training data. This work highlights the potential of combining Generative Artificial Intelligence (GenAI) and multi-agent systems for robust few-shot industrial anomaly detection.