Reinforcement learning-based optimal bipartite formation tracking for uncertain networked agents via enhanced adaptive policy iteration

Optimal bipartite formation tracking of uncertain networked agent systems (NASs) is a hotspot with extensive applications in many fields, and there is an urgent demand for strategies that optimize system performance while ensuring efficiency and stability. With this in mind, this paper proposes a reinforcement learning-based optimal control scheme using an Enhanced Adaptive Policy Iteration (EAPI) algorithm with an adaptive termination mechanism. This scheme enables follower agents to achieve bipartite formation tracking of the leader while optimizing the performance index. Firstly, the definition of the optimal bipartite formation tracking control problem is presented, and the Bellman form of the optimal value function and control law is derived based on the coupled Hamilton-Jacobi-Bellman (HJB) equations. Then, an EAPI algorithm with an adaptive termination mechanism is introduced, which could avoid repeated iterations by setting a termination threshold, thus reducing the computational cost and decreasing the running time without reducing the control performance. Furthermore, the stability, convergence, and optimality of EAPI algorithm are analyzed. Moreover, the optimal control law is approximated and solved through the reinforcement learning framework. Finally, numerical results are conducted to verify the effectiveness of the proposed EAPI-based optimal bipartite formation tracking scheme for NASs.