Reinforcement active learning hierarchical loops

Abstract

A curious agent, be it a robot, animal or human, acts so as to learn as much as possible about itself and its environment. Such an agent can also learn without external supervision, but rather actively probe its surrounding and autonomously induce the relations between its action's effects on the environment and the resulting sensory input. We present a model of hierarchical motor-sensory loops for such an autonomous active learning agent, meaning a model that selects the appropriate action in order to optimize the agent's learning. Furthermore, learning one motor-sensory mapping enables the learning of other mappings, thus increasing the extent and diversity of knowledge and skills, usually in hierarchical manner. Each such loop attempts to optimally learn a specific correlation between the agent's available internal information, e.g. sensory signals and motor efference copies, by finding the action that optimizes that learning. We demonstrate this architecture on the well-studied vibrissae system, and show how sensory-motor loops are actively learnt from the bottom-up, starting with the forward and inverse models of whisker motion and then extending them to object localization. The model predicts transition from free-air whisking that optimally learns the self-generated motor-sensory mapping to touch-induced palpation that optimizes object localization, both observed in naturally behaving rats.