MATADOR: Multimodal traffic accident prediction enhanced by multi-source aggregated emotion recognition

While predicting traffic accidents is challenging, it is highly valuable as it can greatly improve public safety. Previous studies have mostly relied on time-series data capturing drivers’ physiological responses and behavior along with vehicle movement to predict collisions. However, focusing solely on the driver and target vehicle is insufficient, as driving behavior is also influenced by the external environment, especially in emergencies. In this work, we propose a multi-source aggregation model, termed MATADOR. The model aggregates and fuses data from various sources, including multimodal physiological and behavioral indicators of the driver, sensor data from the target vehicle and its surrounding vehicles, and static environmental data such as weather conditions and potential hazards to predict traffic accidents. MATADOR is built upon multi-source feature extraction, multimodal data fusion, and a novel assisting mechanism to detect driver anger, recognize emotion intensity, and predict traffic accident probability over the following 1, 3, and 5 s, respectively. Thus, MATADOR can provide timely alerts to the driver, helping to prevent potential accidents. This proactive approach sets MATADOR apart from previous studies, highlighting its usefulness in real-world applications. Moreover, recognizing the strong link between drivers’ emotional states and accidents, previous studies have utilized multi-task learning to enhance the accuracy of traffic accident prediction. However, they often treat tasks as isolated branches, failing to capture the dependencies between each other. To tackle this challenge, we developed a dynamic assisting mechanism that allows the model to capture the influence of the emotional states of a driver on accident prediction, thereby realizing task-relevance-driven dynamic optimization. Extensive experiments prove that MATADOR significantly outperforms state-of-the-art methods in traffic accident prediction.