Predictive fatigue-aware human-robot collaboration: a real-time, open-source ROS 2 instantiation of a three-module human-centric digital twin framework on a standard cobot.

Abstract

Industry 5.0 cobotics calls for collaborative robots that adapt to the operator's physical and cognitive state in real time. Most current industrial deployments remain reactive, responding only after explicit commands and ignoring the operator's psychophysiological condition. The three-module Human-Centric Digital Twin (HCDT) framework establishes the upstream perception-and-reasoning architecture for such systems using Vision-Language Models, and identifies closed-loop feedback and physical-assistance mechanisms as the natural next layer in the framework's development. Building on that foundation, this paper presents a complementary, real-time instantiation specialised for the resource-constrained, vision-plus-biosignal setting typical of ergonomics-and-safety HRC, deployed on a standard collaborative robot (Universal Robots UR3 with a Robotiq 2F-85 gripper) and extended with closed-loop operator-state adaptation. The Perception module pairs MediaPipe hand-landmark tracking and a Random Forest gesture classifier with Tobii Pro Glasses 3 pupillometry. The Reasoning module combines a ten-state finite-state machine, a weighted composite fatigue score (blink rate, blink duration, PERCLOS, hand-jerk) banded into FRESH, MILD, MODERATE and SEVERE, a saccade-gated commitment fixation, and a Task-Evoked Pupillary Response (TEPR) detector for acute-stress soft-emergency-stop. The Motion module provides a closed-form analytical inverse-kinematics solver, workspace clamping and a slew-rate-limited joint-velocity controller. Velocity-based linear extrapolation of hand position over a 0.5 s horizon enables anticipatory robot pre-positioning. The system is presented as a single-operator feasibility study rather than a generalisation claim. On a leak-free split-then-augment evaluation (520 raw test samples not seen by the training fold in any orientation), the Random Forest classifier reaches 99.62% accuracy with both errors falling in the Okay-Neutral confusion pair that the geometric override layer is designed to catch; a comparison against MLP and KNN baselines on the same split shows all three classifiers within the real-time control budget. Saccade-gated early-commit is designed to recover the modelled 150-300 ms gaze-to-motion lead, and the TEPR Soft E-Stop operates on an independent 300-500 ms channel. End-to-end gesture-to-motion latency has a conservative worst-case ceiling of approximately 830 ms, formed by a camera-buffer term, the 10-frame debounce and one control tick, inside the 1-s interactive response-time limit adopted as the design target. Owing to a Tobii Pro Glasses 3 hardware fault, the physiological pathways (fatigue banding, TEPR soft-stop and gaze commitment) were verified through their downstream control responses to simulated triggers rather than validated on organic pupil data; organic, multi-participant validation is identified as future work.