Multi-trajectory optimization for train using distributional reinforcement learning with conditional value-at-risk

Artificial intelligence methods like reinforcement learning (RL) have been widely studied to train trajectory optimization problems to achieve flexible driving. To meet the demand for flexible driving strategies in actual operations, N optimized trajectories for the single train are usually generated based on different scheduled times. It brings up two issues: the computational cost of N trajectories is N times that of a single trajectory, and manual intervention is required to adjust the initial conditions, such as schedule time. This paper proposes a conditional value-at-risk (CVaR) distributional Q-learning approach (CDQ) to generate trajectories with different driving styles, balancing safety and efficiency. First, analyzing the actual control deviations, the distribution of returns is modeled using the quantile of distributional RL. Then, we introduce CVaR as a risk metric to evaluate the risk of actions and develop risk-sensitive strategies based on various confidence levels, simultaneously optimizing multiple trajectories for the single train. Finally, we simulate the experiments with data from an actual line. The results demonstrate that the CDQ algorithm can simultaneously optimize multiple train trajectories without requiring human intervention. Through a two-layer selection mechanism, five trajectories with varying driving styles can be selected to fulfill scheduling flexibility requirements. Compared to standard Q-learning, distributional Deep Q-Network and other risk-sensitive RL, CDQ shows improved performance in both energy-saving and punctuality. The total computation time of CDQ is only 31.47% and 35.44% of Q-learning and risk-sensitive RL.