Automated analysis of construction safety accident videos using a large multimodal model and graph retrieval-augmented generation

Abstract

Safety investigators are challenged by the manual task of analyzing large volumes of accident videos through the repetitive process of reviewing them frame by frame, which is both tedious and labor-intensive. This paper proposes the Accident Video Analysis framework (AcciVid), which automates this process using a large multimodal model (LMM) integrated with Graph Retrieval Augmented Generation (Graph RAG). Accident video content and regulations are converted into Resource Description Framework (RDF) triples and stored as graphs, enabling regulation-based analysis through Graph RAG. AcciVid detected 90 %p more potential safety violations than human safety investigators, achieving an F2 Score of 82.4 % compared to their F2 Score of 54.8 %. Furthermore, AcciVid required only an average of 42 s to generate a draft report, whereas human safety investigators needed an average of 4.6 h. This demonstrates AcciVid's potential as an assistant to safety investigators in reducing manual workloads while maintaining high accuracy and efficiency.