Beyond algorithms: Utilizing multi-modal emotional and behavioral cues as novel predictors of short-video consumption

The short-video industry has experienced rapid growth in recent years, largely driven by advanced content recommendation systems. While much of the existing research has concentrated on algorithmic improvements, the psychological factors influencing viewing behaviors remain underexplored. This study aims to address this gap by incorporating users' emotional and behavioral indicators into the prediction of short-video viewing behavior. Study 1 was conducted in a controlled laboratory setting, where participants viewed videos on a computer screen while their physiological activity (including electrocardiography and electrodermal activity) was recorded as an objective measure of emotional responses. After viewing the video, participants self-reported their emotions and viewing preferences. Employing a variety of machine learning techniques, we found that both self-reported and physiologically measured emotions were strong predictors of viewing behaviors, with a predictive accuracy exceeding 72 %. Study 2 aimed to enhance ecological validity by having participants view videos on mobile phones, enabling them to swipe between videos as they would in a typical short-video app. Using physiological signals and mobile edge data (including swipe gestures and gyroscope signals), the predictive accuracy for actual viewing behavior reached 82 %. Additionally, a substantial portion of variance in edge signals could be explained by physiological signals. These findings provide valuable insights into the psychological drivers of short video viewing behavior and present a novel, non-intrusive approach to incorporate users’ real-time experiences into content recommendation systems.