Application of unsupervised machine learning methods to actor–critic structures in reinforcement learning for training and online implementation

A fundamental obstacle to the implementation of reinforcement learning (RL) to continuous systems is the large amount of data and training that must take place to achieve a satisfactory control policy. This is exacerbated when the focus is an online implementation. It is the goal of this work to investigate the use of unsupervised learning to make more efficient decisions with the data available, both for learning and exploration in the typical RL algorithm. Gaussian mixture models (GMMs) are used to form a probabilistic prediction for the outcome of proposed actions during exploration, while high-performing data points are subsequently over-sampled to accelerate learning and convergence. With respect to the exploration policy, GMMs are used to predict the outcomes for given actions and used for preventing undesired exploratory actions that can lead to significant loss in control performance and/or violation of safety or other operational constraints. The proposed approach was integrated within a Deep Deterministic Policy Gradient algorithm and was applied to the control of a selective catalytic reduction unit. It was found that a satisfactory policy was found faster and with less overall performance degradation than the standard RL approaches.