Controllable diffusion models for hazardous construction site scene generation

Abstract

Hazardous scene recognition is critical for construction site safety, but the low occurrence of such scenes in real environments leads to insufficient training data, limiting model development. Hazardous scene generation helps address data scarcity but involves complex background-object relationships and significant size differences, making precise layout control difficult. These characteristics make achieving precise layout control challenging for general-purpose hazardous scene generation models. To address these challenges, we propose a novel construction site hazardous scene generation framework based on large language and diffusion models, consisting of a two-stage generation process. In the first stage, we fine-tune the large language model (LLM) through context learning to serve as a text-based layout generator. In the second stage, we introduce a novel text-to-image diffusion model to guide the image generation process, ensuring that the generated image adheres to the scene layout produced in the first stage. Additionally, We propose two key modules, the Layout Enhancement Module and the Scale Fusion Module, to improve image quality and layout adherence. Comparative experiments show that our method generates superior scenes with stronger controllability and higher quality. Testing on a real-world dataset achieved a mAP score of 38.1%, improving model accuracy by 20.4% AP, 17.5% $A{P}_{50}$, and 17.0% AR compared to models trained on real data, demonstrating our method’s effectiveness in hazardous construction site scene generation.