Scalable Large Model for Unlabeled Anomaly Detection With Trio-Attention U-Transformer and Manifold-Learning Siamese Discriminator

Abstract

To identify pattern deviations in large-scale industrial infrastructures, anomaly detection is crucial yet challenging. Previous research has not adequately addressed the characteristics and deployment considerations in these complex scenarios. In this paper, we present InoU, a scalable anomaly detection framework to process unlabeled multivariate time-series data. We incorporate a VAE filter to ease impacts from noisy components in training materials. We propose a scalable trio-attention U-Transformer to construct the typical representation of high-dimensional streams and produce pseudo labels that enable the later training process. The ultra perception and intra-/ inter-flow attention mechanisms are delicately designed to aggregate information from different flows with variable granularities while keeping a global view of the data. Its nested structure helps to maintain high efficiency even when the model is scaled down. We introduce a Siamese discriminator that projects target data into manifolds, and collates discrepancies at the embedding level. This paradigm elevates detection performance far beyond segment-wise error comparison in prior works. We apply contrastive and adversarial learning techniques to optimize manifold projection and detection performance when processing unseen samples. Extensive experiments on five large-scale datasets demonstrate the effectiveness of InoU with an average F1-Score improvement of 5.58%, significantly outperforming the state-of-the-art.