A multimodal deep learning approach for predicting traffic accident severity using crash records, road geometry, and textual descriptions

Abstract

Accurately predicting traffic accident severity is critical for improving road safety management and targeted prevention strategies. This study proposes a novel multimodal deep learning framework integrating structured accident records, detailed road geometry, and unstructured textual descriptions. To our knowledge, this research offers the first large‐scale dataset that combines linear roadway geometry with accident reports for comprehensive severity analysis. The proposed model employs advanced multimodal fusion techniques, including attention‐based gating mechanisms, a mixture‐of‐experts module, and cross‐feature interactions, effectively capturing complex interdependencies among various data modalities. In addition, this paper implements a three‐tiered interpretability analysis at the modality, expert, and feature levels, using SHapley Additive exPlanations values to transparently explain the model predictions. Experimental results demonstrate that our model significantly outperforms traditional and baseline methods, particularly in identifying severe accidents. Interpretability analyses highlight critical insights into accident severity, emphasizing textual descriptions and detailed road characteristics.