Modeling Lead-Lag Structure in Facial Expression Synchrony for Social-Psychological Outcome Prediction from Negotiation Interaction

This study proposes introducing facial-expression synchrony features to machine learning to estimate a customer’s psychological information from online business negotiation dialogue data. It is important for synchrony features to model the information on who led the synchrony and who followed it, the lead-lag structure, because the psychology of the leader and follower can differ. However, conventional synchrony models cannot incorporate such lead-lag structure information because they are based on the assumption that synchrony involves the co-occurrence of features in the same frame. To solve this problem, we propose using synchrony features extracted on the basis of windowed time-lagged cross-correlation, which cuts out a short segment from each of the input sequences and computes the cross-correlation between the segments. Since this method measures the similarity of signals across different frames, it is suitable for modeling the lead-lag structure. We conducted experiments based on an audio visual corpus of business negotiation dialogue assessed with various psychological measurements. The results indicate that considering lead-lag information can improve the accuracy in estimating psychological information.