Artificial Intelligence Can Recognize Whether a Job Applicant Is Selling and/or Lying According to Facial Expressions and Head Movements Much More Correctly Than Human Interviewers

Abstract

Whether an interviewee’s honest and deceptive responses can be detected by the signals of facial expressions in videos has been debated and called to be researched. We developed deep learning models enabled by computer vision to extract the temporal patterns of job applicants’ facial expressions and head movements to identify self-reported honest and deceptive impression management (IM) tactics from video frames in real asynchronous video interviews. A 12- to 15-min video was recorded for each of the N = 121 job applicants as they answered five structured behavioral interview questions. Each applicant completed a survey to self-evaluate their trustworthiness on four IM measures. Additionally, a field experiment was conducted to compare the concurrent validity associated with self-reported IMs between our modeling and human interviewers. Human interviewers’ performance in predicting these IM measures from another subset of 30 videos was obtained by having N = 30 human interviewers evaluate three recordings. Our models explained 91% and 84% of the variance in honest and deceptive IMs, respectively, and showed a stronger correlation with self-reported IM scores compared to human interviewers.