Bimodal Emotion Recognition using Kernel Canonical Correlation Analysis and Multiple Kernel Learning

Abstract

Bimodal emotion recognition on account of kernel canonical correlation analysis (KCCA) and multiple kernel learning (MKL) is investigated and utilized to discover the befitting and effectual fusion pattern with respect to facial expression channel and body gesture channel in the form of video data. Firstly, to relieve calculated quantity of the posterior fusion and classification procedure, the two groups of quondam facial expression and body gesture video data are switched to be indicated as the form of lower dimensional histogram spatio-temporal emotion vectors respectively by Dollar's spatio-temporal feature. Then, KCCA-MKL in the form of multiple kernels is adopted to portray the nonlinear character of facial expression and body gesture video data, and simultaneously to search two modalities' conjunct nonlinear correlative structures by considering the disadvantage of the signal kernel used in KCCA. The rudimentary idea of the KCCA-MKL method is using multiple kernels with the combination of gaussian kernel and $\chi^{2}$ kernel to substitute for the signal kernel in KCCA. In experiment step, some types of the combination of the gaussian kernel and the $\chi^{2}$ kernel are implemented in KCCA-MKL. The test results display that the classification accuracy of the KCCA-MKL approach is 56.91% using the KNN classifier, and is better than two unimodal methods and signal kernel method. Consequently, KCCA-MKL is more unfailing and efficient for bimodal emotion recognition.