AnDR-BLIP2: enhanced semantic understanding framework for industrial image anomaly detection and report generation

Abstract

Nowadays, the rapid development of Large Multimodal Models (LMM) has demonstrated its powerful ability in image understanding. However, when applied to downstream tasks such as industrial anomaly detection, it often lacks competence due to limitations in image parsing ability, pre-training data, and training strategy. Specifically, it struggles with understanding the detailed semantics of abnormal parts of images. As LLM performance continues to improve, the Industrial Image Anomaly Detection Report Generation (IADRG) task may emerge as a new challenge in the future. In this paper, we define the IADRG task as a deeper image understanding task and propose a solution for it. We propose AnDR-BLIP2, a dual-branch multi-modal large model based on the BLIP2 model combined with the SAM visual understanding branch to enhance detailed feature extraction from images. Additionally, we utilize mixed semantic pre-training of general and industrial image data to strengthen the model's ability to understand abnormal content in industrial anomaly detection tasks. Furthermore, our model leverages SAM's pixel-level feature parsing ability to integrate a prompt zero-shot industrial anomaly segmentation method into report generation. Experimental results on Mvtec-AD and VisA datasets demonstrate that our model accurately understands industrial image anomalies and achieves considerable performance in zero-shot anomaly segmentation.