Anomaly Detection for Cyber-Physical Systems Using Transformers

Abstract

Safety and reliability are two critical factors of modern Cyber-Physical Systems (CPS). However, the increasing structural and behavioral complexity of modern automation systems significantly increases the possibility of system errors and failures, which can easily lead to economic loss or even hazardous events. Anomaly Detection (AD) techniques provide a potential solution to this problem, and conventional methods, e.g., Autoregressive Integrated Moving Average model (ARIMA), are no longer the best choice for anomaly detection for modern complex CPS. Recently, Deep Learning (DL) and Machine Learning (ML) anomaly detection methods became more popular, and numerous practical applications have been presented in many industrial scenarios. Most of the modern DL-based anomaly detection methods use the prediction approach and LSTM architecture. The Transformer is a new neural network architecture that outperforms LSTM in natural language processing. In this paper, we show that the Transformer-based deep learning model, which has received much attention recently, can be applied to the anomaly detection of industrial automation systems. Specifically, we collect time-series data from a system of two industrial robots using a Simulink model. Then, we feed these data into our Transformer-based model and train it to be a time-series data predictor. The paper presents the experimental results that show the comparison of precision and speed of a Long-Short Time Memory (LSTM) predictor and our Transformer-based predictor.