A Multistep Reinforcement Learning Control of Shear Flows in Minimal Input–Output Plants Under Large Time-delays

Abstract

Flow control has attracted research for its potential role in reducing drag, suppressing turbulence, and enhancing mixing in fluid systems. The emergence of data-driven modeling and machine learning techniques has sparked new interest in designing control strategies that can adapt in real time to complex, high-dimensional flow environments. However, fluid systems remain particularly challenging testbeds for control design due to their nonlinear and convective nature, which introduces large time delays. In active control, additional difficulties arise from practical constraints, such as the use of localized sensors in limited number. In this work, we investigate a reinforcement learning framework based on a suitable actor–critic algorithm designed to address these challenges. Two test cases representative of transitional shear flows are considered: a linearized version of the Kuramoto–Sivashinsky equation and the control of instabilities in a two-dimensional boundary-layer flow over a flat plate, using a minimal but realistic sensor–actuator configuration. This choice reflects our focus on the limitations that arise from plants of experimental interest. Time delays are identified during a pretraining stage, while the control algorithm employs multistep returns during value iteration. This approach improves both the convergence rate and stability of learning. Furthermore, we show that the look-ahead in the multistep formulation provides a non-trivial beneficial effect in plants where the control task is characterized by a severe credit-assignment issue.