Conditioning multimodal information for smart environments

This study aims at providing signal processing solutions for the conditioning of multimodal information in audio-aided smart camera environments. A novel approach is introduced for processing audio and video within a unified ‘data fusion via fission’ framework. This is achieved using empirical mode decomposition (EMD), a fully data-driven algorithm which facilitates analysis at multiple time-frequency scales. Its adaptive nature makes it suitable for processing real-world data and allows, for example, signal conditioning (denoising, illumination invariant video) and robust feature extraction. Furthermore, complex extension of the EMD algorithm are used to quantify shared dynamics between the conditioned modalities facilitating multimodal fusion. The proposed collaborative approach is used to model human-human interaction.