Acted sleepy speech vs. real sleepy speech: Human perception and machine prediction for sleepiness estimation

Abstract

As agentic AI systems increasingly operate in high-stakes human-centered environments, their ability to detect physiological states such as sleepiness is critical for mitigating health and safety risks. Accurately estimating sleepiness from speech is essential for managing these risks, including adverse health outcomes, reduced productivity, and occupational safety hazards. For example, chronic sleep deprivation increases the risk of cardiovascular disease and impairs cognitive performance. Therefore, accurately determining sleepiness holds significant value for both health monitoring and workplace risk management. Since speech serves as a primary interface for human-AI interaction, estimating sleepiness from speech has emerged as an efficient way to mitigate these risks through AI. However, current machine learning (ML) studies have struggled to estimate sleepiness from speech, showing weak Spearman correlations (below 0.40) with perceived sleepiness levels (ground-truth sleepiness). As ML performance depends on high-quality training data, obtaining speech from noticeably sleepy individuals is essential for its improvement. However, collecting such data is risky and costly, as it requires physiologically sleep-deprived speakers with at least 24 h of prolonged wakefulness. Given that humans can mimic sleepy speech, this paper proposes that acted sleepy speech can be an effective surrogate for modeling sleepiness patterns. Our study demonstrates two key findings: First, human listeners rated acted sleepy speech as significantly sleepier than real sleepy speech obtained from a 25-hour sleep deprivation protocol, as confirmed by Welch’s t-test. Second, ML models trained on acted speech significantly outperformed those trained on real sleepy speech, achieving a 0.57 correlation with perceived sleepiness levels and 0.83 accuracy in detecting cognitive impairments in sleep-deprived individuals awake for 25 h, even with fewer samples and different lexical transcripts, indicating robustness to lexical variation. This human-centered, ethical, and efficient approach demonstrates that acted sleepy speech can advance real-world speech-based sleepiness estimation and cognitive impairment detection systems. Ultimately, it offers a promising pathway to real-world solutions for reducing sleepiness-related risks in the workplace, improving health outcomes, and enhancing the ability of human-centered agentic AI to support daily human tasks.