Automatic perception of potential safety hazards: A cross-modal multi-task framework for feature alignment, image classification and captioning

Automatic perception of potential safety hazards (PSHs) is critical for ensuring workplace safety and protecting property against significant threats. PSHs perception involves determining whether hazards exist, capturing on-site images, and completing inspection reports, which are critical for mitigating these risks. Though computer vision techniques like image classification and image captioning offer promising alternatives for PSHs perception. However, comprehensive hazard perception requires not only hazard identification but also semantic relationship comprehension among scene entities to formulate descriptive safety reports. To address the multifaceted nature of PSHs perception, this study proposes a cross-modal multi-task learning (MTL) method named Hazard-MTL, which jointly optimizes three synergistic tasks: feature alignment (image–text), binary image classification, and image captioning. Specifically, our approach employs a scene graph-guided chain-of-thought data augmentation method that integrates knowledge prompts and multi-task contextual reasoning to produce semantically coherent and informationally complete risk descriptions. To improve the model robustness, a bidirectional contrastive loss was designed to suppress irrelevant cross-modal similarities. Additionally, a dynamic joint training strategy is introduced that combines progressive teacher forcing with adaptive loss weighting to achieve harmonized multi-task optimization. Our model outperforms single-task baselines with 72.7% $F_1$ score (+10.7%) for PSHs classification and 1.575 CIDEr (+0.619) for description generation. Hazard-MTL advances holistic scene understanding by integrating MTL, offering a safer automated solution for enterprise and construction safety management.