Understanding Human Internal States: I Know Who You Are and What You Think

Abstract

For the successful interaction between human and machine agents, the agents need understand both explicitly-presented human intention and unpresented human mind. Although the current human-agent interaction (HAI) systems mainly rely on the former with keystrokes, speech, and gestures, the latter will play an important role for the new and up-coming HAIs. In this talk we will present our continuing efforts to understand unpresented human mind, which may reside at the internal states of neural networks in human brain and may be estimated from brain-related signals such as fMRI (functional Magnetic Resonance Imaging), EEG (Electroencephalography), and eye movements. We hypothesized that the space of brain internal states have several independent axes, of which temporal dynamics have different time scales. Special emphasis was given to human memory, trustworthiness, and sympathy to others during interactions. Human memory changes much slowly in time, and is different from person to person. Therefore, by analyzing brain-related signals from many stimulating images, it may be possible to identify a person. On the other hand the sympathy to others has much shorter time constants during human-agent interactions, and may be identified for each user interaction. The trustworthiness to others may have slightly longer time constants, and may be accumulated by temporal integration during sequential interactions. Therefore, we measured brain-related signals during sequential Theory-of-Mind (ToM) games. Also, the effects of human-like cues of the agents to the trustworthiness were evaluated. At this moment the estimation of human internal states utilizes brain-related signals such as fMRI, EEG, and eye movements. In the future the classification systems of human internal states will be trained with audio-visual signals only, and the current study will provide near-ground-truth labels.