S2R-CMI: A Robust Deep Reinforcement Learning method based on counterfactual estimation and state importance evaluation under additive noise disturbance

Abstract

The development of Deep Reinforcement Learning (DRL) overcomes the limitations of traditional reinforcement learning in discrete spaces, extending its applications to scenarios in continuous spaces. However, disturbance commonly encountered in real-world environments poses significant threats to the performance of DRL algorithms, potentially leading to erroneous decision-making by agents and severe consequences. To address this issue, this paper proposes a Robust Deep Reinforcement Learning (RDRL) method named S2R-CMI, which aims to mitigate the impact of additive noise in the state space on DRL performance without requiring prior knowledge of the noise disturbance. Specifically, a state-based and reward-based conditional mutual information mechanism is designed to dynamically capture state importance and estimate its contribution to rewards. To address the lack of counterfactual data during training, a counterfactual label estimation method is proposed to approximate the counterfactual reward distribution while avoiding local optima during network training. State importance is then evaluated to quantify the impact of disturbance on states. Finally, we validate the proposed method in five scenarios: Cartpole, LunarLander-Discrete, LunarLander-Continuous, Build-Marine, and Half-Cheetah, under state disturbance. Experimental results demonstrate that S2R-CMI significantly enhances the robustness of DRL algorithms. Furthermore, we conduct experiments in some scenarios without state disturbance, and the results indicate that the method also achieves strong performance, further verifying its superiority and generalization capabilities.