Data efficient online learning of robot behaviours via qualitative planning and reinforcement learning

Autonomous robots execute complex behaviours to perform tasks in real-world environments. Reinforcement learning can acquire such behaviours, however, often requires a large number of iterations to reach an operational behaviour. This makes it inefficient for online learning, that is, learning on board the robot as it operates. Combinations of techniques such as model-based reinforcement learning, planning, and behavioural cloning, attempt to narrow the search space of trial-and-error learning. However, they rely on a significant degree of domain knowledge. We develop a domain independent Data Efficient Planning and Learning Architecture for online skill acquisition and which is applied to locomotion tasks on a multi-tracked robot typical of those designed for urban search and rescue. We build a qualitative model of the robot’s dynamics from online behavioural traces, that trades accuracy for domain independence in elevating the skill acquisition problem into a symbolic representation. Then a forward-chaining planner finds an operational sequence of qualitative symbolic actions enabling the robot to complete a task, from which quantitative action parameters representing the robot’s actuator movements are extracted. The qualitative plan places constraints on valid parameter values. This enables online reinforcement learning to refine the parameters into satisficing (or optimal) actuator movements, making trial-and-error learning data efficient in terms of the number of trials. By applying our architecture in a “closed-loop”, the qualitative model is improved from the reinforcement learning trials, refining the final robot’s operation, along with discovering new emergent behaviours.