Curiosity-Driven Reinforced Learning of Undesired Actions in Autonomous Intelligent Agents

Abstract

Autonomous exploring agents are encouraged to explore unknown states in an environment when equipped with an intrinsic motivating factor such as curiosity. Although intrinsic motivation is a useful mechanism for an autonomous exploring agent in an environment that provides sparse rewards, it doubles as a mechanism for causing the agents to act in undesirable ways. In this paper, we show that highly-curious agents, attached with neural networks trained with the Machine Learning Agent Toolkit's (ML-Agents) implementation of the Proximal Policy Optimization (PPO) algorithm, and Intrinsic Curiosity Module (ICM), learn undesirable or reckless behaviors relatively early in the training process. We also show that strong correlations in the PPO training statistics of misbehaving agents may indicate when an actual human should intervene for safety during the RL training process.