Query-guided multimodal learning network for anomaly detection underneath high-speed trains

Anomaly detection on the bottom of high-speed trains is crucial for train safety. However, the complexity and variability of anomalies, along with the intricate environment in which they occur, pose significant challenges to timely detection. To address this issue, we propose the Query-guided Multimodal Learning Network (QMLNet) that exploits multimodal information to discover anomalies. Specifically, in QMLNet, the CNN-Transformer Feature Fusion (CFF) Module uses queries to guide the learning and fusion of multi-level visual features, enriching the expressiveness of features at each level for improved prediction of anomaly masks. The Attention-based Mask Refinement (AMR) module generates masks based on the attention mechanism to enhance the features, learns the features using queries, and obtains a global representation of the features at different levels, which will be used along with the textual features for better prediction of the anomaly categories. Compared to state-of-the-art methods, the proposed method achieves superior results on our anomaly dataset, leading by a significant margin. In addition, our method achieves the best performance on the public defect detection dataset, the VISION dataset, significantly outperforming most methods, which proves the generalization and robustness of our approach.