Multimodal Emotion Detection based on Visual and Thermal Source Fusion

The contactless emotion detection is an interesting research topic today. In this paper, we first study the physiological basis of human emotions to better understand what happens in our body when emotions arise and change. We then introduce the interconnection between the brain trunk vessels and the facial vessels. The investigation reveals that the variations of human emotions could be reflected by facial blood horizontal flow, and the detection of facial blood horizontal flow could be realized by mainly measuring the Remote photoplethysmography (rPPG) and gray scale variation on human cheeks. To validate these findings, we set up an emotional evoking experiment to capture the RGB and thermal videos of human testees, extract out horizontal facial blood flows, and finally classify these features into three different emotions (i.e., fear, happiness and sadness) by learning. The reported classification accuracy reaches 0.841, based on total 45 testees.