Uncertainty weighted policy optimization based on Bayesian approximation

Abstract

Efficient exploration remains a major challenge in the field of reinforcement learning (RL). Bayesian methods have been widely investigated within the RL paradigm and are used to implement intelligent exploration strategies. However, most of these methods inevitably introduce some complexity within the Bayesian neural networks (BNNs) or are difficult to optimize elegantly. In this work, a novel algorithm called uncertainty weighted policy optimization (UWPO) based on Bayesian approximation, is introduced. UWPO theoretically analyzes the uncertainty of the policy space using the Dirichlet distribution and Monte Carlo (MC) dropout for both discrete and continuous spaces, eliminating the need for an explicit distribution representation in BNNs. The algorithm also proposes an implicit distributional training method for the value function, which is compatible with Bayesian inference. Moreover, an uncertainty-weighted update principle is adopted to adaptively adjust the contribution of each training instance to the objective. Finally, comparing UWPO with other prevailing deep reinforcement learning (DRL) algorithms on the Atari, MuJoCo, and Box2D platforms. The experimental results demonstrate that the algorithm improves the average reward score by nearly 15% while reducing computational costs by 20% compared to current state-of-the-art methods.