Frame-by-frame processing-based robot system integrating multimodal body language feature recognition and fusion for thermal comfort monitoring

Real-time thermal comfort estimation is crucial for managing occupant well-being in intelligent indoor environments. This study proposes a multimodal body language-integrated real-time thermal comfort monitoring (MRTM) system to dynamically estimate the thermal comfort distribution of multiple occupants within a built space. To realize dynamic multimodal feature fusion in complex scenes with multiple persons, we developed a frame-by-frame multibody language feature recognition module for cross-frame posture association based on SlowFast networks. We combined the module with Fanger’s predicted mean vote model to dynamically match environmental parameters and individual characteristics, generating a three-dimensional PMV mesh model. The system was experimentally validated in a university research institute lobby 143.2 m² in Shenzhen, China, using occupant flow data collected from students and staff total of 1269 exhibiting diverse behaviors such as sitting, standing, and walking. The MRTM system achieved high accuracy rates in sex, emotion, clothing, and behavior recognition 95.14 %, 96.64 %, 92.15 %, and 96.26 %, respectively. The system attained a ROC-AUC score of 0.874 in real-time estimation of occupants’ thermal comfort, thereby providing evidence of its robustness and effectiveness in analyzing thermal comfort within the environment.