Average-Reward Baseline Centering: A Plug-In Stabilizer for DDPG and MADDPG in C-V2X Platooning

Abstract

Fast and reliable data exchange is essential for connected and autonomous vehicles, particularly in platooning scenarios where vehicles travel in close formation and depend on timely updates for safety and traffic efficiency. Actor–critic reinforcement learning can exhibit unstable learning signals and suboptimal policies under dynamic vehicular network conditions, especially in continuing (non-episodic) resource-allocation tasks, where slot-level fast fading, bursty CAM arrivals, and mode-switching between V2I and V2V links amplify reward variance. We propose a plug-and-play Baseline-Centered Target (BCT) update for DDPG-family methods. The core idea is: before forming each temporal-difference target, a sliding-window estimate of the long-run average reward is subtracted from the sampled reward, with no other change to the training loop. Theoretically, BCT preserves the fixed point of the Bellman equation and reduces TD target variance without biasing the actor update. BCT is evaluated across four coordination paradigms—centralized DDPG independent-learning DDPG (IL-DDPG), CTDE MADDPG, and Hybrid MADDPG—each paired with its BCT variant, with TD3 as a single-agent reference. BCT variants consistently improve steady-state global reward across all paradigms ($+11.5\%$ to $+26.0\%$) and produce more stable training curves. In the Centralized paradigm, BCT and TD3 achieve comparable gains ($+17.9\%$ and $+19.0\%$ respectively), confirming complementary. In the Independent paradigm, IL-TD3 degrades below its base while BCT-IL-DDPG still improves by $+11.5\%$, demonstrating BCT’s robustness under multi-agent non-stationarity. Results are validated across multiple random seeds and window sizes ($W \in \lbrace 5,\ldots,20\rbrace$), and an $N_{\text{RB}}$ ablation ($\lbrace 3, 6\rbrace$) confirms BCT-Hybrid retains its advantage, with mean AoI dropping 83% at $N_{\text{RB}}=6$.