RADAR: Reactive and Deliberative Adaptive Reasoning - Learning When to Think Fast and When to Think Slow

Abstract

When designing and deploying Reinforcement Learning (RL) algorithms, one typically selects a single value for the discount rate, which results in an agent that will always be equally reactive or deliberative. However, similarly to humans, RL agents can benefit from adapting their planning horizon to the current context. To enable this, we propose a novel algorithm: RADAR: Reactive and Deliberate Adaptive Reasoning. RADAR enables an agent to learn to adaptively choose a level of deliberation and reactivity according to the state it is in, given that there are cases where one mode of operation is better than the other. Through experiments in a grid world, we verify that the RADAR agent has the capability to adapt its reasoning modality to the current context. In addition, we observe that the RADAR agent exhibits different preferences regarding its thinking modes when a penalty for mental effort is included in its mathematical formulation.