Keyframes selection from multiscene videos for stress detection

Abstract

In the modern world, stress is a rising global issue that impacts both human physical and mental health. Early stress detection is vital for timely intervention and prevention of health decline. Although widely deployed video cameras in surroundings offer a contact-free channel for stress detection, the computational cost is exceedingly high compared with the methods based on physiological and linguistic signals. To use multiscene videos cost-efficiently, we propose a fine-grained two-stage keyframe selection framework for efficient stress detection. The first emotion-oriented keyframe selection stage intends to reduce irrelevant and redundant frames per video owing to the high frame rate. The second stress-oriented keyframes selection stage aims to grasp emotion dynamics reflecting one’s stressful states, expecting to achieve a decent effect with fewer frames through peer-attended, collaborative deep reinforcement learning. The performance analysis on the developed dataset highlights the benefits of our two-stage multiscene collaborative keyframe selection process for stress detection, achieving an accuracy of 83.61% and an F1-score of 83.48% in three-labeled stress detection and an accuracy of 71.80% and an F1-score of 66.78% in five-labeled stress detection, with a frame selection rate of 0.14% per video. Implications and further possible improvements are discussed at the end of the paper.