Coupled flows as guidance for model-based policy optimization

Abstract

Model-based reinforcement learning (MBRL) offers high sample efficiency but suffers from cumulative multi-step prediction errors that degrade long-term performance. To address this, we propose a coupled flows-guided policy optimization framework, where two coupled flows quantify and minimize the discrepancy between the true and learned state–action distributions. By reducing this divergence, the loss functions serve as both a discriminator, selecting more accurate rollouts for policy learning, and a reward signal, refining the dynamics model to mitigate multi-step errors. Theoretical analysis establishes a bound on the expected return discrepancy. Empirical evaluations demonstrate that our method achieves higher cumulative rewards than the representative model-based approaches across diverse control tasks. This highlights its applicability in data-scarce domains such as robotics, recommendation systems, and autonomous driving.