Incorporating multi-path risk assessment in transformer-based pedestrian crossing action prediction

Abstract

This paper proposes a Transformer-based framework for predicting pedestrian crossing actions that uses visualized pedestrian-vehicle collision risks, which are assessed from multiple potential paths. Our framework contains two sequential steps: (1) multi-path risks of a pedestrian-vehicle interaction (PVIs) at each time point are estimated and encoded into an RGB image, which captures a high-density array of safety information. (2) a multi-modal fusion architecture that incorporates both risk images and historical sequential data (e.g., pedestrian action and vehicle velocity) is developed based on the Cross-Attention Transformer. The model outputs are also risk-informed, categorized as yielding, risky crossing, and safe crossing. Experiments are conducted on real-world data from the Euro-PVI dataset. Through two-dimensional mapping tests, risk images are validated to have significant spatiotemporal feature differences and transition associations under different PVIs. The Transformer architecture proves to be an effective method for processing multi-path risk images. Prediction accuracy reaches 87.34% for short-term forecasts (0.5 s ahead), maintains stability as the prediction time horizon progressively extends to 2 s, and improves the prediction of abrupt action switches. For further exploration and validation, the risk image data and imaging code are available at www.github.com/Sivan0227/PVI-Risk-Image.