Combining Speech Features for Aggression Detection Using Deep Neural Networks

Predicting the intensity level of aggression is a challenging problem in surveillance applications. Since there are no trivial fusion rules or classifiers, we developed a fusion framework to accomplish this complex task using Deep Neural Networks. This framework used a low level that contains the audio-visual features, an intermediate level composed of a set of concepts (meta-features) and a high level which is a final evaluation of the multimodal aggression detection. In this paper, we study the prediction of multimodal level for aggression detection and both Context and Semantics meta-features. This prediction is based on the audio modality using sensor and semantic information. Using meta-features for the semantic part of speech, we show the added value of such extra-information on the fusion process when the situations are more complicated. We also propose to use different text-based features such as linguistic and word affect features that will provide significant results for predicting the two meta-features and the multimodal aggression level using Deep Neural Networks when they are fused with the acoustic features although the nature of spontaneous speech.