Enhancing visual-LLM for construction site safety compliance via prompt engineering and Bi-stage retrieval-augmented generation

The escalating frequency of safety incidents on construction sites requires an effective safety management framework. Traditional computer vision systems are constrained by their static nature, limited generalization, and inadequate semantic comprehension. This paper integrates a multi-modal Visual Language Model (VLM) with our proposed Bi-stage Retrieval-Augmented Generation (BiRAG) framework, which enhances safety compliance monitoring based on construction site images, with high scalability and adaptiveness to evolving safety standards without tedious model fine-tuning. A TriPhased prompt (TPP) and a decision-tree-based compliance judgment prompt are designed to enhance the VLM's ability to interpret worker behaviors and safety compliance from site images. A context-aware chunking strategy and hybrid retrieval algorithm are developed to improve the analysis against relevant safety regulations. Experiments with images collected from a real construction site in Hong Kong demonstrated a 7.73 % increase in retrieval accuracy and an 11.66 % improvement in compliance analysis accuracy, offering a holistic construction safety management solution.