3D-MMFN: Multi-level multimodal fusion network for 3D industrial image anomaly detection

Abstract

3D-based image anomaly detection (AD) is a crucial computer vision task in industrial manufacturing. Most existing methods only focus on 2D shape-based detections. However, there is still limited research for detecting anomalies in 3D shapes using multimodal features. Some existing techniques developed for this task are mostly unsuitable for industrial defect detection for several reasons. Firstly, they rely mostly on memory banks, resulting in high storage overheads, making them difficult to deploy on production lines. Secondly, the multimodal features, in the existing 3D industrial AD algorithms, are concatenated directly which cause a significant disruption between the features and degrades the detection efficiency. Thirdly, their inference speed is not fast enough to achieve real-time detection. To address these challenges, we propose a deployment-friendly network named 3D-MMFN. Our model comprises of the following components: (1) The pre-trained feature extractors that generate local features from multi-stream inputs of RGB images, surface normal maps, and point clouds. (2) A novel point-to-pixel based fusion module that efficiently fuses multi-level multimodal features to mitigate disruption during the fusion operation. (3) An anomaly generator module that generates anomalous features from normal multimodal fused features, enabling self-supervised training of 3D-MMFN while eliminating the need for extensive memory banks. Experimental results on the MVTec3D-AD and Eyecandies dataset demonstrate the effectiveness of our proposed model, showcasing significant performance improvements over state-of-the-art methods.