RuleNet: rule-priority-aware multi-agent trajectory prediction in ambiguous traffic scenarios

Accurately predicting surrounding traffic participants’ intentions and future trajectories is essential for automated vehicles in interactive scenarios. These interactions often involve diverse semantic interpretations embedded within different traffic rules. Accordingly, learning the priority characteristics of traffic rules offers a promising pathway to improving prediction performance. However, traffic rules are frequently ambiguous and are overlooked by trajectory prediction models. To address this issue, this paper introduces RuleNet, a multi-agent trajectory prediction framework that incorporates the priority evaluation of ambiguous traffic rules. RuleNet consists of three primary components. First, built upon Graph Neural Networks, it extracts agent kinematics, road topology, and traffic rule representations. Next, a multi-attention mechanism is employed to model interactions among agents, between historical and predicted trajectories, and across different prediction modes, thereby generating initial trajectory proposals. Finally, a rule-guided refinement module is introduced to adjust the predictions in accordance with learned rule priorities. This study focuses on two key traffic rule categories: safety and right-of-way, which are quantified using time to collision and relative distance, depending on the interaction type. Rule priorities are calculated through Signal Temporal Logic robustness measures and integrated into the prediction refinement process. Comprehensive experiments on the INTERACTION dataset validate the effectiveness of RuleNet. Results show that it outperforms existing baselines, achieving a 1.8–5.1% increase in prediction accuracy and a 16.8% improvement in safety. Furthermore, ablation studies are conducted to examine the influence of individual rule types and fusion strategies on model performance. The findings highlight three main findings: 1) Distance-based rules considerably improve prediction accuracy in ambiguous intersection scenarios. 2) Temporal rules are more influential in interactions involving vulnerable road users than in vehicle-to-vehicle cases. 3) Integrating rule priorities into both feature extraction and attention mechanisms yields the best overall performance.