Twin Trust Region Policy Optimization

Abstract

Trust region policy optimization (TRPO) is an iterative reinforcement learning algorithm that both maximizes a surrogate objective based on a generalized advantage function and enforces a trust region constraint between two consecutive policies based on the Kullback-Leibler divergence in each iteration. On the one hand, its surrogate objective only approximates the positive theoretical improvement and thus relaxes the theoretical guarantee. One the other hand, currently, TRPO is approximately solved via a linear approximation for its objective and a quadratic approximation for its constraint, which results in an upper bound for the linear search of the step size. However, this solution strategy does not provide a lower bound for the step size search. Intuitively, a lower bound could be determined manually, but such a setting has no physical meaning. To this end, we present an alternative solution in this article, which is to exchange the objective and the constraint in TRPO according to reciprocal optimization technique, and then derive a reciprocal TRPO. Applying a similar approximation solution of TRPO to our reciprocal TRPO produces a lower bound for the step size search that has a physical interpretation for improving the surrogate objective or return. Further, we aggregate the original TRPO with this reciprocal TRPO to construct a twin TRPO, whose step size has a lower bound from our reciprocal TRPO and an upper bound from TRPO, to facilitate the policy optimization and achieve a least return improvement. Extensive experiments on twelve benchmark environments show that our twin TRPO is superior to six existing techniques: the original TRPO, proximal policy optimization, off-policy TRPO and two entropy regularized TRPOs, as well as our reciprocal TRPO. Our code is available at https://github.com/HTXu-UTS/twinTRPO.