Q-Cogni: An Integrated Causal Reinforcement Learning Framework

Abstract

We present Q-Cogni , an algorithmically integrated causal reinforcement learning framework that redesigns Q-Learning to improve the learning process with causal inference. Q-Cogni achieves improved policy quality and learning efficiency with a prelearned structural causal model of the environment, queried to guide the policy learning process with an understanding of cause-and-effect relationships in a state-action space. By doing so, we not only leverage the sample efficient techniques of reinforcement learning but also enable reasoning about a broader set of policies and bring higher degrees of interpretability to decisions made by the reinforcement learning agent. We apply Q-Cogni on vehicle routing problem (VRP) environments including a real-world dataset of taxis in New York City using the Taxi and Limousine Commission trip record data. We show Q-Cogni's capability to achieve an optimally guaranteed policy (total trip distance) in 76% of the cases when comparing to shortest-path-search methods and outperforming (shorter distances) state-of-the-art reinforcement learning algorithms in 66% of cases. Additionally, since Q-Cogni does not require a complete global map, we show that it can start efficiently routing with partial information and improve as more data is collected, such as traffic disruptions and changes in destination, making it ideal for deployment in real-world dynamic settings.