{"title":"Deep Reinforcement Learning Empowered Sign Language Recognition for Enhanced Accessibility in Smart Home","authors":"Elakkiya R","doi":"10.1109/THMS.2026.3695755","DOIUrl":"https://doi.org/10.1109/THMS.2026.3695755","url":null,"abstract":"This article introduces reinforcement learning-based adaptive sign language recognition (RLASLR), a framework designed to enhance sign language recognition (SLR) in smart home environments. RLASLR integrates deep reinforcement learning (DRL) to refine interpretation accuracy through four reinforcement-guided modules: the adaptive spatial fusion network (ASFNet); the spatial attention refinement engine; the semantic contextual enhancement mechanism; and the temporal consistency alignment network. These modules collectively optimize feature diversity, refine attention on salient sub-actions, enrich semantic context, and maintain temporal coherence. The deployment strategy adopts an actor–critic method, enabling RLASLR to adapt dynamically to user–environment feedback and changing smart home conditions. The novelty of this article lies in the custom reward function that jointly weights spatial, attentional, semantic, and temporal components, together with the validation of DRL-based sign language recognition (SLR) under diverse smart home conditions. Additionally, the study introduces new evaluation metrics such as Feature Diversity Enhancement, fusion mechanism adaptability, spatial attention map adjustment, feature relevance score and attention map consistency for a fine-grained assessment of robustness. Experimental results on the ISLAN and ISL-CSLR datasets demonstrate that RLASLR achieves superior accuracy and resilience to signer variability, lighting changes, and occlusion. These contributions mark a significant step toward practical, adaptive, and user-centric assistive technologies for individuals with hearing impairments.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 4","pages":"753-763"},"PeriodicalIF":4.4,"publicationDate":"2026-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148694973","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Enhancing Early Mortality Prediction With Clinical Notes as Time-Series Scores","authors":"Gabriel Vázquez-Serrano;Maite Oronoz;Alicia Pérez","doi":"10.1109/THMS.2026.3697894","DOIUrl":"https://doi.org/10.1109/THMS.2026.3697894","url":null,"abstract":"The optimization of early mortality prediction models to accurately identify high-risk patients during a hospital admission is key in medical research to assist in healthcare decision-making. While these models use structured (physiological features) or unstructured (free-text clinical notes) data, integrating them synergistically, especially for early prediction, remains a challenge. We introduce a novel, model-agnostic approach to represent clinical notes as quantitative, task-specific time-series, structurally aligning them with physiological features. Instead of relying on complex multimodal architectures, we shift fusion from the architecture level to the data level by transforming each note into a numerical mortality risk score, creating a new time-series feature that can be seamlessly integrated into any standard time-series prediction model. We evaluate our approach on the following: first, early prediction, second, data scalability, and third, versatility across centralized, federated, and local learning frameworks, in an attempt to assess the stability of the approach in scenarios with different amounts of data to learn from. Using only readily accessible data from the first 8–24 h postadmission, our method significantly boosts baseline model performance, with relative improvements of up to 7 % in area under the receiver operating characteristic curve and 28 % in area under the precision–recall curve over prior work on early prediction that relies solely on physiological data. Moreover, we find that incorporating our text-derived scores significantly achieves higher performance gains than nearly doubling the training data size in scenarios where no textual information is considered. Rather than proposing a predictive model, we present a new technique to structure clinical narratives, enabling a more effective fusion with temporal, quantitative health data for improved early prediction of mortality.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 4","pages":"764-773"},"PeriodicalIF":4.4,"publicationDate":"2026-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=11561745","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148695004","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"CNDNet: A Cross-State Neural Decoding Network for Unified Semantic Decoding From Perceptual and Imagery EEG","authors":"Jinze Tong;Wanzhong Chen","doi":"10.1109/THMS.2026.3696743","DOIUrl":"https://doi.org/10.1109/THMS.2026.3696743","url":null,"abstract":"Electroencephalography (EEG), owing to its noninvasiveness, high temporal resolution, and the central role of perception and imagery in human cognition, has made EEG-based neural decoding highly significant in fields such as cognitive science and brain–computer interfaces. However, most existing studies focus solely on semantic decoding from either visual perception or visual imagery EEG, without considering their shared neural circuits and semantic invariance. As a result, the two types of EEG have not been placed within a unified framework, greatly limiting the potential of current decoding methods. To address this challenge, a Cross-State Neural Decoding Network (CNDNet) is proposed, where the semantic discrepancy between the two types of EEG signals is minimized as a training objective, enabling the extraction of essential semantic features and realization of cross-state semantic decoding. First, a Feature Extractor based on a spatial–frequency convolutional architecture is employed to extract spatial–spectral features, with a weighted spatial attention mechanism used to enhance semantic components. Then, a Category Classifier combining Group Normalization and Kolmogorov–Arnold Networks is applied to ignore differences in semantic feature representations and capture invariant semantics. Finally, a Domain Discriminator with a similar structure to the Category Classifier is used to determine the domain category of features. CNDNet, validated on two public datasets, achieves an accuracy gain of up to 16.60 percentage points over leading models, demonstrating efficient and robust cross-state neural decoding.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 4","pages":"887-896"},"PeriodicalIF":4.4,"publicationDate":"2026-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148694783","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Decision Behavior of Leading Vehicle Driver With Predictive-Forward-Collision-Warning Considering Group Heterogeneity","authors":"Qiang Fu;Xiaohua Zhao;Chen Chen;Wenhao Ren","doi":"10.1109/THMS.2026.3691608","DOIUrl":"https://doi.org/10.1109/THMS.2026.3691608","url":null,"abstract":"The decision behavior of the leading vehicle driver significantly impacts the safety of the connected mixed platoon, especially on the longitudinal decision during emergency braking events. However, driving abilities vary among leading vehicle drivers. Predictive-forward-collision-warning (PFCW) is an emerging technology that enhances driving safety by providing traffic information beyond the driver’s line of sight. But the influence of PFCW on their decision behavior remains unclear. Therefore, it is necessary to investigate the leading driving competence of heterogeneous driver groups under the influence of PFCW. This study established a test platform for the connected mixed platoon using driving simulation technology. Subsequently, it developed a connected human–machine interface incorporating PFCW functionality and recruited 36 participants to experiment. The experiment aimed to analyze decision behaviors during the emergency braking event involving preceding vehicles. The latent profile analysis model was employed to identify heterogeneous driver groups. Moreover, the cumulative prospect theory was used to characterize drivers’ decision behaviors. The results indicated that PFCW encouraged drivers to maintain a larger distance from the preceding vehicle and to exhibit smoother braking behavior. However, the group of older, skilled drivers who were subjectively concerned about the driving status of following vehicles exhibited higher driving risk as the leading vehicle driver when PFCW was absent. Overall, this study enhances the understanding of longitudinal decision behavioral performance among heterogeneous driver groups in the presence of PFCW. It offers insights for subsequent customization of the selection criteria for leading vehicle drivers, and the tailored design of PFCW based on the characteristics and behavioral preferences of these heterogeneous driver groups.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 4","pages":"826-835"},"PeriodicalIF":4.4,"publicationDate":"2026-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148695053","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Advancing Eating Intention Inference for Autonomous Feeding Robots With Deep Networks Empowered by Enhanced Attention Mechanism","authors":"Xuan Zhang;Guanzhong Zhou;Hailong Huang","doi":"10.1109/THMS.2026.3689126","DOIUrl":"https://doi.org/10.1109/THMS.2026.3689126","url":null,"abstract":"The development of physically assistive robots capable of providing eating assistance holds tremendous potential in improving the quality of life for seniors and individuals with disabilities. However, autonomous feeding depends on accurate eating intention inference. Unfortunately, most existing methods fall short in investigating human intention and the dynamic variability of human behavior during eating. This article proposes an eating intention inference method for assistive robots in human–robot interaction by combining the Residual Network (ResNet) and long short-term memory (LSTM), facilitating autonomous feeding by robotic arms. Taking inspiration from observable facial movements during eating, we extract crucial features, including facial landmarks and the amplitude of the chin-to-nose distance computed from them, which clearly reflects human eating intention. Considering the noise and data corruption present in the input sequences, we employ the Gaussian function as the convolutional kernel within the ResNet architecture. Furthermore, we incorporate an aggregated variance attention mechanism in the LSTM hidden layer to better capture the dynamic changes in the input data. Experimental results demonstrate the strong performance of our method, achieving an accuracy of 98.0% on a self-collected dataset and outperforming other approaches. Real-world experiments with a robotic arm and an RGB camera verify the efficacy and real-time predictive performance of the proposed method.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 4","pages":"733-742"},"PeriodicalIF":4.4,"publicationDate":"2026-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148694930","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Alexis Gomez-Hernandez;Mobina Amrollahi;Elham Entezarizarch;Jundi Liu;Jing Yang
{"title":"A Systematic Review of Human-in-the-Loop Adaptive Systems: Design, Implications, and Future Directions","authors":"Alexis Gomez-Hernandez;Mobina Amrollahi;Elham Entezarizarch;Jundi Liu;Jing Yang","doi":"10.1109/THMS.2026.3691217","DOIUrl":"https://doi.org/10.1109/THMS.2026.3691217","url":null,"abstract":"Adaptive systems aim to address the human-out-of-the-loop problem in automation, associated with reduced situation awareness and users' overreliance. These systems modify their behavior based on system, task, or operator-based triggers, with the latter offering a human-centered approach by responding to different human states such as mental workload or fatigue. Despite growing interest in adaptive systems, no review has consolidated empirical findings. To fill this gap, and using PRISMA methodology, 34 articles were included after full-text screening. The analysis focuses on the types of human states measured, the sensors and modeling techniques used, the nature of the adaptive interventions, and their reported outcomes. Cognitive states were the most commonly assessed, often using EEG due to its ability to capture brain activity in real time. Physical state, frequently applied in rehabilitation, relied on EMG and IMU sensors to assess muscle activation and joint accuracy. Machine learning methods were more prevalent than rule-based systems, likely due to their flexibility and robustness. While function allocation emerged as the dominant adaptive strategy, system performance outcomes varied. Future work should prioritize user-centered design, avoiding task disruption or disengagement. Finally, standardized evaluation frameworks are required to assess human–machine performance.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 4","pages":"836-847"},"PeriodicalIF":4.4,"publicationDate":"2026-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148694962","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Fixed-Time Inverse Reinforcement Learning Based Human–Machine Autonomous Game Control","authors":"Mi Wang;Lingling Lv;Huaicheng Yan;Huai-Ning Wu","doi":"10.1109/THMS.2026.3695135","DOIUrl":"https://doi.org/10.1109/THMS.2026.3695135","url":null,"abstract":"With the expansion of artificial intelligence (AI) from information intelligence to physical intelligence (embodied intelligence), human–machine collaborative intelligence has become a key path to achieving strong AI. This article proposes an autonomous game control algorithm based on fixed-time inverse reinforcement learning to address the limitations of existing human–machine collaboration methods, such as switching control and sharing control, in terms of deep collaboration and dynamic optimization. This algorithm innovatively integrates two core mechanisms: first, a fixed-time parameter identification law is designed to estimate the human control strategy online, breaking through the asymptotic convergence limitation of traditional behavior learning algorithms. Furthermore, an autonomous game control law is designed to achieve real-time decision optimization on the machine side. By combining Lyapunov stability theory with IRL techniques, this method ensures that the system converges to near Nash equilibrium within a fixed-time interval, significantly improving the timeliness and stability of human–machine collaboration. Finally, a simulation example is provided to verify the validity of the proposed game control algorithm.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 4","pages":"858-867"},"PeriodicalIF":4.4,"publicationDate":"2026-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148694997","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"IoT-Integrated Augmented Reality for Spatially Aware Human–Machine Interaction for IEEE Transactions on Human-Machine Systems","authors":"Akhil Ajikumar;Mohsen Moghaddam","doi":"10.1109/THMS.2026.3696828","DOIUrl":"https://doi.org/10.1109/THMS.2026.3696828","url":null,"abstract":"A human–machine interface (HMI) is an essential component of any modern industrial machine, acting as the portal for human workers to setup, monitor, control, and troubleshoot machines. Yet, existing HMIs often pose challenges in usability and learnability due to overwhelming data, nonstandard interfaces, and physical design limitations. We argue that these challenges stem from a lack of spatiotemporal alignment of data and controls with the machine, leading to information overload and loss as the operator repeatedly translates signals between the HMI and machine. We propose a novel augmented reality (AR) interface integrated with Internet of Things (IoT), which affords spatially aware, natural mapping of real-time data and controls with the machine while providing visibility and immediate feedback. We build the IoT-integrated AR system through a Kepware-ThingWorx-Vuforia pipeline implemented in a model cyber-physical factory for cellphone assembly. We conducted a between-subjects user study with 20 participants split into two groups, who operated three assembly line stations using our system deployed on HoloLens 2 and the machine's built-in Siemens HMIs in opposite orders. The findings show the significant effects of our AR interfaces on improving efficiency, lowering task completion times, reducing errors, and enhancing usability and mental load. We conclude the paper by discussing the limitations and several directions for future research on IoT-integrate AR interfaces to transform how humans interact with industrial machines.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 4","pages":"723-732"},"PeriodicalIF":4.4,"publicationDate":"2026-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148694791","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Phy-HHMR: Physics-Aware Holistic Human Mesh Reconstruction","authors":"Junjie Cai;Binhong Ye;Lei Wang;Baoyu Liu;Xiaoliang Ma;Jun Cheng","doi":"10.1109/THMS.2026.3701774","DOIUrl":"https://doi.org/10.1109/THMS.2026.3701774","url":null,"abstract":"In recent years, single-image-based 3-D human modeling has become a promising research direction, with potential applications in augmented reality, human–computer interaction. However, 3-D human body modeling from a single image often produces physically unreasonable results in close-interaction scenarios due to severe occlusion and depth ambiguity. We propose a physics-aware framework for holistic human mesh recovery (Phy-HHMR), a physics-driven holistic reconstruction system that recovers expressive 3-D meshes within a unified framework. Our method features an interaction-aware optimization pipeline, combining physical consistency priors with spatially guided diffusion refinement. By employing geometric gap regularization and tokenized motion memory, the system ensures nonpenetration and contact plausibility, maintaining kinematic accuracy and a physical foundation. Phy-HHMR leverages a system-level joint optimization strategy and paired-flow couplers to sustain overall mechanical consistency. Simulation results show that our method achieves state-of-the-art performance and stronger robustness to reconstruction defects, providing a reliable basis for virtual reality and human–computer collaboration.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 4","pages":"784-793"},"PeriodicalIF":4.4,"publicationDate":"2026-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148694839","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Human-in-the-Loop Adaptive-Enhanced Constraint Management for Switched Systems via Self-Adjusting Feedback Triggering Mechanism","authors":"Lei Liu;Yujie Cui;Ruonan Ren;Tong Wang","doi":"10.1109/THMS.2026.3701733","DOIUrl":"https://doi.org/10.1109/THMS.2026.3701733","url":null,"abstract":"Human-in-the-loop switched systems often suffer from unmeasurable states and strict safety constraints, which may lead to unacceptable workload or even catastrophic errors for human operators. This article proposes an adaptive-enhanced constraint-management strategy for such human-machine shared-control platforms. The main results and contributions are as follows. first, a self-adjusting event-triggered mechanism is introduced to reduce the update frequency of the autopilot, alleviating the operator’s cognitive burden while ensuring communication efficiency; Second, an adaptive state observer is designed to reconstruct unmeasurable states in real time, providing accurate state information for feedback control; Third, by incorporating a tan-type barrier Lyapunov function, we theoretically guarantee that all state variables strictly remain within predefined human-safe ranges throughout system operation; fourth, fuzzy-logic systems are employed to approximate unknown nonlinearities arising from pilot-autopilot interactions, with established approximation error bounds; Finally, under the average dwell-time switching law, we rigorously prove that the closed-loop human–machine system achieves uniformly ultimately bounded stability while excluding Zeno behavior. Numerical simulation results validate the effectiveness of the proposed approach, demonstrating improved tracking accuracy and reduced control update frequency compared with traditional periodic control schemes.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 4","pages":"927-936"},"PeriodicalIF":4.4,"publicationDate":"2026-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148694854","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}