Zero-shot framework for construction equipment task monitoring

Vision-based monitoring of construction equipment is limited in scalability due to the high resource demands of collecting and labeling large datasets across diverse environments. This study proposes a framework that employs Zero-Shot Learning (ZSL) and Multimodal Large Language Model (MLLM) to recognize construction equipment tasks from video frames without additional training data. The framework operates in two stages: (i) a zero-shot construction equipment detection stage that includes detection and tracking modules and (ii) an MLLM-based monitoring stage, utilizing the proprietary model (i.e., GPT-4o mini) to recognize tasks. Experiments showed that the framework achieved an F1-score of 82.2% for equipment detection using ZSL. A Multiple Choice Question (MCQ) dataset was constructed for evaluating MLLM, which achieved an accuracy of 79.0%. A practical case study, focusing on excavator tasks, demonstrated accurate recognition of both idle states and complex operations. These results highlight the proposed framework's potential to automate construction equipment monitoring.