Discovering neural policies to drive behaviour by integrating deep reinforcement learning agents with biological neural networks

Abstract

Deep reinforcement learning (RL) has been successful in a variety of domains but has not yet been directly used to learn biological tasks by interacting with a living nervous system. As proof of principle, we show how to create such a hybrid system trained on a target-finding task. Using optogenetics, we interfaced the nervous system of the nematode Caenorhabditis elegans with a deep RL agent. Agents adapted to strikingly different sites of neural integration and learned site-specific activations to guide animals towards a target, including in cases where agents interfaced with sets of neurons with previously uncharacterized responses to optogenetic modulation. Agents were analysed by plotting their learned policies to understand how different sets of neurons were used to guide movement. Further, the animal and agent generalized to new environments using the same learned policies in food-search tasks, showing that the system achieved cooperative computation rather than the agent acting as a controller for a soft robot. Our system demonstrates that deep RL is a viable tool both for learning how neural circuits can produce goal-directed behaviour and for improving biologically relevant behaviour in a flexible way. Deep reinforcement learning (RL) has been successful in many fields but has not been used to directly improve behaviours by interfacing with living nervous systems. Li et al. present a framework that integrates deep RL agents with the nervous system of the nematode Caenorhabditis elegans. Their study shows that trained agents can assist animals in biologically relevant tasks and can be studied after training to map out effective neural policies.