Optimizing large vision-language models for context-aware construction safety assessment

Abstract

This paper presents a context-aware large vision-language model (LVLM) for automated construction site safety assessment, addressing the limitations of existing models in domain-specific hazard recognition. It introduces a framework that combines domain-specific image-text data generation, vision encoder fine-tuning for improved object recognition, and Low-Rank Adaptation (LoRA)-based model adjustment for context-aware safety reasoning. The model was evaluated on 400 images from 10 hazardous situations, demonstrating superior performance in the image captioning task (average ROUGE-L: 0.3852, SPICE: 0.3615, SBERT-based similarity: 0.7484). For safety assessment, the fine-tuned model achieved 94.25 % accuracy in predicting safety status, significantly outperforming GPT-4 V (53.25 %) and LLaVA 1.5 (48 %). The quality of textual justifications was assessed using both GPT-4 V-based and expert-based evaluations of relevance and preference. In both settings, the fine-tuned model received the highest scores, demonstrating robust and context-aware safety reasoning. These findings confirm that domain-specific fine-tuning enhances safety classification and hazard interpretation, advancing construction site monitoring.