TMAN: A temporal multimodal attention network for backchannel detection

Backchannel responses play an essential role in human communication, which are often expressed by listeners to show their attention and engagement to speakers without interrupting their speech. Their automatic detection is crucial for developing conversational AI agents that engage in human-like, responsive communication. Backchanneling can be conveyed via a combination of various non-verbal cues, such as head nodding and facial expressions. However, these cues are often subtle, brief and sparse during conversations, posing significant challenge in the accurate detection of backchannel responses. This study introduces TMAN, a sequential three-stage multimodal temporal network designed to effectively encode behavioral features from four human visual modalities. It incorporates three attention modules to encode subtle “micro” actions, such as specific gestures or facial expressions, that occur at each frame, as well as temporal “macro” behavior patterns, such as sustained body and head movements, into a final representation for backchannel detection. These are often expressed in backchannel responses, thereby enhancing the detection capabilities. Comprehensive experiments conducted on two public datasets demonstrate that TMAN significantly enhances performance and achieves state-of-the-art results. Extensive ablation studies validate the contribution of each attention module and visual modality employed in our model, and identify the appropriate feature transformation and implementation setups for effective backchannel detection. An in-depth investigation of the model inference process further demonstrates the effectiveness of TMAN attention modules, particularly in processing both “micro” and temporal “macro” behavior patterns in multimodal visual cues.