Multimodal manifold learning using kernel interpolation along geodesic paths

Abstract

In this paper, we present a new spectral analysis and a low-dimensional embedding of two aligned multimodal datasets. Our approach combines manifold learning with the Riemannian geometry of symmetric and positive-definite (SPD) matrices. Manifold learning typically includes the spectral analysis of a single kernel matrix corresponding to a single dataset or a concatenation of several datasets. Here, we use the Riemannian geometry of SPD matrices to devise an interpolation scheme for combining two kernel matrices corresponding to two, possibly multimodal, datasets. We study the way the spectra of the kernels change along geodesic paths on the manifold of SPD matrices. We show that this change enables us, in a purely unsupervised manner, to derive an informative spectral representation of the relations between the two datasets. Based on this representation, we propose a new multimodal manifold learning method. We showcase the performance of the proposed spectral representation and manifold learning method using both simulations and real-measured data from multi-sensor industrial condition monitoring and artificial olfaction. We demonstrate that the proposed method achieves superior results compared to several baselines in terms of the truncated Dirichlet energy.