End2You

Abstract

Multimodal profiling is a fundamental component towards a complete interaction between human and machine. This is an important task for intelligent systems as they can automatically sense and adapt their responses according to the human behavior. The last 10 years, several advancements have been accomplished with the use of Deep Neural Networks (DNNs) in several areas including but not limited to affect recognition[1,2]. Convolution and recurrent neural networks are core components of DNNs that have been extensively used to extract robust spatial and temporal features, accordingly. To this end, we introduce End2You[3] an open-source toolkit implemented in Python and based on Tensorflow. It provides capabilities to train and evaluate models in an end-to-end manner, i.e., using raw input. It supports input from raw audio, visual, physiological or other types of information, and the output can be of an arbitrary representation, for either classification or regression tasks. Well known audio- and visual-model implementations are provided including ResNet[4], and MobileNet[5]. It can also capture the temporal dynamics in the signal, utilizing recurrent neural networks such as Long Short-Term Memory (LSTM). The toolkit also provides pretrained unimodal and multimodal models for the emotion recognition task using the RECOLA dataset[6]. To our knowledge, this is the first toolkit that provides generic end-to-end learning for profiling capabilities in either unimodal or multimodal cases. We depict results of the toolkit on the RECOLA dataset and show how it can be used on different datasets.