Modular Reinforcement Learning for a Quadrotor UAV With Decoupled Yaw Control

Abstract

This letter presents modular reinforcement learning (RL) frameworks for the low-level control of a quadrotor, with direct control of yawing motion. While traditional monolithic RL approaches have been successfully applied to real-world autonomous flight, they often struggle to precisely control both translational and yawing motions due to their distinct dynamic characteristics and coupling. Moreover, training a large-scale monolithic network typically requires extensive training data to achieve broad generalization. To address these issues, we decompose the quadrotor dynamics into translational and yaw subsystems and assign a dedicated modular RL agent to each. This design significantly improves performance, as each RL agent is trained for its specific purpose and integrated in a synergistic way. It further enhances robustness, as potential failures within one module have minimal impact on the other, promoting fault tolerance. These improvements are demonstrated through flight experiments achieved via zero-shot sim-to-real transfer, where it is shown that the proposed modular policies substantially enhance training efficiency, tracking performance, and adaptability to real-world conditions.