Data-driven safety management of worker-equipment interactions using visual relationship detection and semantic analysis

Abstract

Existing technologies struggle to accurately identify interactions between workers and equipment, as well as the deep semantics of complex construction scenes. To address these limitations, this paper proposes an automated construction site safety management system designed to enhance scene understanding and identify safety hazards while focusing on hazard-area and personal protective equipment (PPE) interaction. The system transforms image information into worker-centric triplets and generates precise textual descriptions through semantic enhancement, enabling effective scene analysis. By comparing the generated descriptions with predefined hazard statements, the system identifies potential risks. Experimental results demonstrate a 9.6 % improvement in recall for Ng-mR@K metrics (K = 20, 50, 100). Additionally, the system successfully filters over 90 % of invalid relationships, achieving 83.7 % accuracy in semantic similarity matching, significantly enhancing detection precision and semantic understanding. By advancing from object detection to a structured image-to-triplet-to-text framework, this paper offers an efficient and reliable solution for automated construction site safety management.