Channel-Specific Stimulus Sensitivity, Reliability, and Complementarity of Multimodal Wearable Sensing During Short-Form Influencer Video Exposure.

Abstract

Multimodal wearable sensing is increasingly used to quantify audience engagement with short-form video, yet it remains unclear which sensing channels actually carry stimulus-driven information at the exposure durations typical of this format. We recorded electrodermal activity, photoplethysmographic pulse, triaxial acceleration, video-based eye tracking, and automated facial expression analysis from 50 young adults while they viewed 20 short-form influencer clips (5.6-19.3 s) presented in individually randomized order, yielding 1000 participant × clip trials and 12 primary indices spanning four modalities. Using crossed random-effects variance partitioning, split-half reliability with Spearman-Brown projection, permutation-based multivariate testing, participant-wise decoding, and inter-subject correlation, we show that stimulus-driven variance is strongly channel-specific: it reaches 13.7% for fixation rate and 10.2% for facial engagement but only 0.11% for pulse rate and 0.23% for RMSSD. Clip-level estimates were highly reliable for facial and oculomotor indices (r=0.82-0.94) and poor for autonomic indices (r=0.34-0.58), which would require 134-370 participants to reach ρ=0.80. Between-modality dependence was almost six times weaker than within-modality dependence, and five components were retained by parallel analysis. Oculomotor and facial features jointly identified individual clips at 16.8% balanced accuracy against a 5% chance level, whereas electrodermal and pulse features, taken separately, did not exceed chance. Electrodermal responses were nonetheless stimulus-locked in 18 of 20 clips. A per-second re-analysis of nine channels showed that the ordering partly reverses for within-clip dynamics: tonic electrodermal and pupil time courses were reliable across viewers (ρ50=0.88 and 0.93), facial time courses were not (ρ50≤0.69), and pulse rate was unreliable under both treatments. We derive explicit design guidance for sensor selection and sample size in short-exposure multimodal engagement studies.