{"title":"Inferring Pilot Cognitive States Using a Multimodal Physiology-Based Graph Fusion Model","authors":"Yiran Xu;Yilin Li;Shuai Wu;Xuefeng Men;W. Tong","doi":"10.1109/THMS.2026.3675080","DOIUrl":"https://doi.org/10.1109/THMS.2026.3675080","url":null,"abstract":"Accurate recognition of pilots' cognitive states is essential for intelligent cockpit systems operating in complex and high-load flight environments. Existing methods either rely on single-modal physiological signals or fail to fully exploit the intrinsic temporal, spatial, and cross-modal structures of multimodal data. To address these limitations, we propose a multimodal physiology-based graph fusion (MPGFusion) network, an end-to-end deep framework that explicitly models the hierarchical graph structure and spatiotemporal dependencies of multimodal physiological signals. Unlike conventional approaches that depend on hand-crafted features or independent modality-specific encoders, MPGFusion directly takes raw multimodal physiological time series as input and jointly captures first, multiscale temporal dynamics via parallel temporal kernels, second, intramodality spatial relationships through single-modal graphs, and third, intermodality heterogeneous interactions using a multimodal graph integrated with graph transformation network, graph convolutional network, and gated recurrent unit. This hierarchical graph-based design enables structured cross-modal feature interaction while preserving modality-specific characteristics. Experiments conducted on the CogPilot dataset demonstrate that MPGFusion significantly outperforms representative baselines, achieving 97.49% classification accuracy. Extensive ablation studies validate the contribution of each architectural component and highlight the complementary nature of multimodal physiological signals. The proposed framework provides a generalizable and effective solution for multimodal cognitive state recognition in aviation scenarios.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 3","pages":"615-624"},"PeriodicalIF":4.4,"publicationDate":"2026-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148504920","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jiwon W. Kim;Kexin Wang;Michael C. Dorneich;Eliot Winer;Lori J. Brown;Geoffrey Whitehurst
{"title":"Evaluating the Effectiveness of Educational Augmented Reality Authoring Tool: Insights From Instructors, Experts, and Students","authors":"Jiwon W. Kim;Kexin Wang;Michael C. Dorneich;Eliot Winer;Lori J. Brown;Geoffrey Whitehurst","doi":"10.1109/THMS.2026.3667033","DOIUrl":"https://doi.org/10.1109/THMS.2026.3667033","url":null,"abstract":"This article assessed the effectiveness of an augmented reality (AR) authoring tool for aviation weather training developed to support flight instructors without requiring technical expertise. While AR-based training showed promise in aviation weather education, there are limited tools available for nontechnologist instructors. To address this challenge, an instructor-centered AR authoring tool was developed and evaluated using a multiphase framework. The AR authoring approach was validated on three dimensions: instructors’ authoring experience; instructional quality of produced AR lessons; and students’ learning outcomes and engagement. Flight instructors successfully developed interactive AR lessons on aviation weather topics of their choice, and reported high usability and manageable workload for authoring. Instructors reported increased confidence in creating AR content, while their concerns regarding technical complexity, mental effort, and time demands decreased. Subject matter experts rated the instructional quality of instructor-authored AR lessons highly. Using the produced AR lessons, students showed significant gains in knowledge and motivation. These findings demonstrate the potential of instructor-centered authoring approach to enable the creation of high-quality, interactive educational content without requiring technical expertise, and to provide a replicable, evidence-based framework for assessing educational technologies and supporting broader integration of AR in education.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 3","pages":"572-581"},"PeriodicalIF":4.4,"publicationDate":"2026-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148504306","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Driver's Perceived Risk Prediction Method Based on Semisupervised Learning Strategy","authors":"Siwei Huang;Chenhao Yang;Chuan Hu;Xi Zhang;Ronghui Zhang","doi":"10.1109/THMS.2026.3667782","DOIUrl":"https://doi.org/10.1109/THMS.2026.3667782","url":null,"abstract":"Safety has become a critical issue in the development of automated driving systems (ADSs). Drivers' perception of risk determines their acceptance, trust, and use of ADS. However, driver's perceived risk is subjective and difficult to evaluate using traditional risk assessment methods. To address this issue, a driver's subjective perceived risk (DSPR) model is proposed, which regards driver's perceived risk as a dynamically triggered mechanism that exhibits anisotropy and attenuation. Subsequently, 20 participants are recruited for a driver-in-the-loop experiment to report their real-time subjective risk ratings (SRRs) when experiencing various real-world automated driving scenarios. A convolutional neural network and bidirectional long short-term memory with temporal pattern attention (CNN-Bi-LSTM-TPA) network is embedded into a semisupervised learning strategy to predict driver's SRRs, aiming to reduce data noise caused by the subjective randomness of drivers. Results illustrate that our proposed DSPR model combined with the semisupervised learning strategy achieves the highest prediction accuracy of 87.91% in predicting driver's SRRs, outperforming other three state-of-the-art risk models. Compared to the model trained on the original data, the semisupervised method improves accuracy by 20.12%. The proposed CNN-Bi-LSTM-TPA network presents the highest among four different network structures. Finally, a genetic algorithm is applied to optimize parameters, which improves the accuracy to 89.95%. This study offers an effective method for assessing driver's perceived risk, providing support for the safety enhancement of ADS and driver's trust improvement.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 3","pages":"625-635"},"PeriodicalIF":4.4,"publicationDate":"2026-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148504926","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Hanru Zhou;Dongjun Liu;Gang Xie;Tianyang Qin;Kehan Wang;Wanzeng Kong
{"title":"Cognition-Guided Coupled Learning for Camouflaged Object Detection","authors":"Hanru Zhou;Dongjun Liu;Gang Xie;Tianyang Qin;Kehan Wang;Wanzeng Kong","doi":"10.1109/THMS.2026.3671297","DOIUrl":"https://doi.org/10.1109/THMS.2026.3671297","url":null,"abstract":"Camouflaged object detection (COD) aims to identify targets that visually blend into their surroundings, posing significant challenges due to low contrast between targets and backgrounds, diverse target appearances, and limited annotated training samples. A critical yet often overlooked subtask in COD is object presence detection, which determines whether a camouflaged object actually exists in an image. This step is essential as it underpins accurate localization and segmentation and enables timely decisions in practical applications such as disaster rescue and wildlife monitoring. However, most existing methods assume the target is always present and focus mainly on localization, limiting their robustness and reliability in complex scenarios. Inspired by the human brain's remarkable ability to perceive camouflaged objects by integrating edge and contextual cues, we propose a cognitive-visual coupling learning framework to enhance object presence detection. Specifically, electroencephalogram (EEG) signals are incorporated during training to embed human cognitive patterns into the model's visual feature learning through coupled representation learning. Notably, during inference, our framework relies solely on visual input, effectively leveraging cognitive insights acquired during training to improve detection performance. Extensive experiments demonstrate that our approach achieves superior accuracy compared to state-of-the-art methods and exhibits strong generalization across diverse visual backbone architectures and unseen cross-dataset scenarios.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 3","pages":"513-522"},"PeriodicalIF":4.4,"publicationDate":"2026-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148506019","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Locating Motionless Human in Through-Wall NLOS Scenarios With a Commercial mmWave Device","authors":"Zhongkai Deng;Qiliang Yang;Jianchun Xing;Bowei Feng;Wenjie Chen;Qizhen Zhou","doi":"10.1109/THMS.2025.3633627","DOIUrl":"https://doi.org/10.1109/THMS.2025.3633627","url":null,"abstract":"Indoor localization technology is essential for enabling location-based services (LBS). Millimeter-wave (mmWave) radar offers the potential for high-precision LBS; however, accurately localizing stationary individuals or those located behind obstacles remains a challenge due to the weakness and susceptibility of crucial echo signals. We propose a feasible scheme for locating motionless human in indoor through-wall nonline-of-sight (NLOS) scenarios, named LoMo, which infers accurate locations of a target from physiological signals. To achieve this goal, we design a coherent accumulation method to improve the target signal-to-noise ratio and strengthen the expression of weak signals. Then, we troubleshoot the coherent multipath problem by introducing an enhanced super-resolution algorithm with Toeplitz reconstruction of the covariance matrix, which enables the accurate separation of location-related signals. Finally, to avoid the ambiguity caused by diverse environmental variables, we present an adaptive clustering method for mmWave point clouds that considers the spatial features of the scene, allowing the obtaining of real location information without intensive manual effort. To validate the feasibility of LoMo, we run extensive experiments covering human diversity, orientations, and obstacle types. Experimental results show that LoMo achieves a localization error within 17 cm, and a correct identification rate of 96.1% within ±30 cm under through-wall NLOS conditions, outperforming state-of-the-art methods in comparison.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 3","pages":"602-614"},"PeriodicalIF":4.4,"publicationDate":"2026-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148505645","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Huayang Wu;Yilin Lang;Qinyuan Ren;Etienne Burdet;Yanan Li
{"title":"Differential Game With Motor Intent Prediction for Diverging Human Motion Plan","authors":"Huayang Wu;Yilin Lang;Qinyuan Ren;Etienne Burdet;Yanan Li","doi":"10.1109/THMS.2026.3677525","DOIUrl":"https://doi.org/10.1109/THMS.2026.3677525","url":null,"abstract":"When a human controls a robot directly or via teleoperation, incomplete information and unpredictable environmental conditions can lead to conflicts between their plans. Differential game theory (GT) offers a framework for optimal robotic assistance, but existing methods for identifying the human model require a shared plan. This article introduces an approach to deal with a diverging human plan by leveraging a neuromechanical model of their viscoelasticity to directly estimate their motion intent during movement. The viscoelastic gains can then be integrated into a GT framework to compute optimal contribution of the robot to the common motor task. We evaluated the proposed method in experiments, comparing it with fixed impedance control and nonoptimal variable impedance control. Results demonstrate stable interactions even in the presence of conflicting motion plans, and superior performance relative to these two alternative methods.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 3","pages":"469-478"},"PeriodicalIF":4.4,"publicationDate":"2026-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148506017","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Workload Does Not Work: On the Many Problems of the NASA-TLX","authors":"Skye S. Taylor;Matthew L. Bolton","doi":"10.1109/THMS.2026.3664262","DOIUrl":"https://doi.org/10.1109/THMS.2026.3664262","url":null,"abstract":"The NASA-Task Load Index (TLX) and its variants are the most widely used measures of mental workload. However, an increasing body of literature suggests that there are critical problems with this measure. This includes confusing elements of its design that may bias human response or result in administrator error, inconsistent dimensions, limitations in how dimensions are weighted when computing overall workload scores, multicollinearity between dimensions biasing results, and fundamental measurement theory problems that impact the meaningfulness of computed overall workload. This article provides an overview of these problems. We then make recommendations for addressing them through changes in experimental practice and design, as well as practices that can be used by researchers in the intervening period. We recommend that researchers assess the appropriateness of the TLX for every study, combine its use with other measures, analyze dimensions separately instead of overall workload, and use within-subject designs when possible. Finally, we call on the research community to standardize NASA-TLX versions and instructions, audit existing TLX results, and investigate methods to address existential issues.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 3","pages":"682-687"},"PeriodicalIF":4.4,"publicationDate":"2026-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148171636","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Improving Driver Emotion Recognition Using Structure-Detail-Aware Blind Super-Resolution","authors":"Wei-Yen Hsu;Yi-Chun Chiu;Hsin-Yun Chang","doi":"10.1109/THMS.2026.3681915","DOIUrl":"https://doi.org/10.1109/THMS.2026.3681915","url":null,"abstract":"In intelligent driving environments, accurate driver emotion recognition is critical for enhancing human-machine interaction and ensuring road safety. However, due to the constraints of real-world driving conditions, images captured by in-vehicle cameras frequently suffer from blurring, noise, low illumination, and reduced resolution. These visual degradations significantly impair the performance of emotion recognition systems. To address these challenges, we propose a novel structure preservation and detail recovery blind super-resolution (SpDrBSR) framework specifically designed for driver monitoring in human-machine systems. We design the novel cross-domain structure-detail-aware transformer (CSDFormer), which effectively integrates the intradomain and cross-domain information to achieve global and local structure preservation and detail recovery. Specifically, we first apply Canny edge detection to ensure precise edge localization and structural preservation. Building on this, the proposed CSDFormer leverages long-range intradomain and cross-domain dependencies through two key modules: the cross-domain fusion module, which integrates domain features to maintain structure and detail, and the global mixed cross-domain attention module, which retrieves global information, refines similarity scores, and filters irrelevant signals. A cross-attention mechanism further enhances feature interaction and complementarity, enabling accurate reconstruction of complex image structures. We conduct experiments on real-world datasets to verify the effectiveness of the proposed method. The results indicate that the proposed SpDrBSR outperforms the state-of-the-art approaches in terms of quantitative metrics and achieves better structure preservation and detail recovery than those approaches in terms of visual quality. Furthermore, the results also indicate that the proposed SpDrBSR can significantly enhance driver emotion recognition accuracy by nearly 8%.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 3","pages":"672-681"},"PeriodicalIF":4.4,"publicationDate":"2026-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148504314","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Enhancing Navigation Through Natural Sensory Mappings: A Cross-Modal Approach","authors":"Pingping Jiang;Christopher Kent;Jonathan Rossiter","doi":"10.1109/THMS.2026.3668079","DOIUrl":"https://doi.org/10.1109/THMS.2026.3668079","url":null,"abstract":"Individuals with visual impairments frequently depend on assistive navigation tools to live independent lives. Sensory substitution devices (SSDs) can provide navigational assistance through advancements in human sensory processing, hardware, and algorithms. In this article, we incorporate psychological mechanisms into SSD design to facilitate intuitive and measurable cross-modal translation for assisted navigation in unknown spaces. We developed LightWave, a device that translates visual distance into frequency (via touch or sound) using a cross-modal mapping function derived from psychophysical research, which demonstrates a natural correlation between visual distance and frequency across auditory and tactile modalities. Blindfolded participants navigated cluttered environments using either a traditional cane, auditory frequency cues, or tactile frequency cues. Key performance metrics, including walking distance, time, and collisions, were measured to compare effectiveness across devices. The results show that both auditory and tactile frequency navigation outperformed the traditional cane, achieving near-100% success rates compared to the cane’s 85% . Participants using the audio and tactile devices covered an average of 13 m between start and end points, compared to 18.05 m with the cane, and completed navigation faster, taking 54.8 s with audio and 50.95 s with tactile, versus 74.09 s with the cane. These findings show the potential for improving SSDs by incorporating cross-modal psychological principles, demonstrating in LightWave a novel approach to enhance personal navigation through natural intuitive feedback mechanisms.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 3","pages":"523-531"},"PeriodicalIF":4.4,"publicationDate":"2026-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148504312","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"An Educational Human Machine Interface Providing Request-to-Intervene Trigger and Reason Explanation for Enhancing the Driver’s Comprehension of ADS’s System Limitations","authors":"Ryuji Matsuo;Hailong Liu;Toshihiro Hiraoka;Takahiro Wada","doi":"10.1109/THMS.2026.3679748","DOIUrl":"https://doi.org/10.1109/THMS.2026.3679748","url":null,"abstract":"Level 3 automated driving systems (ADSs) have attracted significant attention and are being commercialized. A level 3 ADS prompts the driver to take control by issuing a request to intervene (RtI) when its operational design domains (ODDs) are exceeded. However, complex traffic situations can cause drivers to perceive multiple potential triggers of RtI simultaneously, causing hesitation or confusion during take-over. Therefore, drivers need to clearly understand the ADS’s system limitations to ensure safe take-over. This study proposes a voice-based educational human machine interface (HMI) for providing RtI trigger cues and reasons to help drivers understand ADS’s system limitations. The results of a between-group experiment using a driving simulator showed that incorporating effective trigger cues and reason into the RtI was related to improved driver comprehension of the ADS’s system limitations. Moreover, most participants, instructed via the proposed method, could proactively take over control of the ADS in cases, where RtI fails; meanwhile, their number of collisions was lower compared with the other RtI HMI conditions. Therefore, using the proposed method to continually enhance the driver’s understanding of the system limitations of ADS through the proposed method is associated with safer and more effective real-time interactions with ADS.","PeriodicalId":48916,"journal":{"name":"IEEE Transactions on Human-Machine Systems","volume":"56 3","pages":"650-661"},"PeriodicalIF":4.4,"publicationDate":"2026-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148505035","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}