A soft actor-critic reinforcement learning-based method for remaining useful life prediction

Abstract

Remaining useful life (RUL) prediction techniques play a crucial role in manufacturing equipment condition management and maintenance planning. Currently, data-driven deep learning methods have made significant advancements in this field. However, traditional approaches have not adequately considered the temporal correlations in both sensor data and RUL prediction values during the degradation process of equipment. The existing reinforcement learning (RL) methods face challenges such as lacking of sufficient lifespan variation information in the state variables, ignorance of dynamic changes in prediction error in the reward function design, and adoption of fixed interaction termination conditions that can't effectively promote the agent's learning of device degradation information. Therefore, this paper proposes a RL model based on the soft actor-critic (SAC) algorithm. Firstly, an autoencoder is employed to extract key features from the data collected by sensors. Subsequently, these key features, along with multi-dimensional lifespan features containing information from multiple historical time steps, are utilized to construct the state variables in RL. Next, a reward function is formulated taking into account error gradients. Finally, a progressive early stopping method is proposed to train the model. Extensive experiments are conducted on the CMAPSS dataset and XJTU-SY bearing dataset, and the proposed method demonstrates higher prediction accuracy compared to mainstream approaches.