Prospect utility with hyperbolic tangent function

Abstract

One branch of safety in reinforcement learning is through integrating risk sensitivity within the Markov Decision Process framework. The objective is to mitigate low-probability events that could lead to severe negative outcomes. Eliminating such risky events is usually done by incorporating a utility function on the expected return; therefore, reshaping the reward structures according to the risk levels associated with different outcomes. The temporal difference learning algorithm can be modified with a utility to capture risk. Notably, such utility functions are either convex or concave depending on the desired risk behavior. Given the outcome space and depending on the risk-sensitivity mode, concave utilities may promote risk-averse behavior and convex utilities may encourage risk-seeking strategies. Such function structure is demonstrated in Prospect Theory, and this motivates a novel formulation using the hyperbolic tangent function called PTanh. Using PTanh, experiments are performed to assess the effect of the diminishing marginal property on the risk-averse policies. It is concluded that there is a correlation between the marginal and selecting the risk-averse parameters. The marginals influence the effectiveness of the averse policies. When the marginals are considered, PTanh can demonstrate better results in terms of a ratio of average reward per prohibited state rate. Furthermore, using empirical evidence, the policy experiments shown with PTanh generalize to other utilities of the Prospect Shape.