Methods of Multi-Modal Data Exploration

Abstract

Techniques and tools designed for information retrieval, data exploration or data analytical tasks are based on the relational and text-search model, and cannot be easily applied to unstructured data such as images or videos. Researcher communities have been trying to reveal the semantics of multimedia in the last decades with ever-improving results in various tasks, dominated by the latest success of deep learning. Limits of object retrieval models drive the need for data exploration methods that support multi-modal data, like multimedia surrounded by structured attributes. In this paper, we describe, implement and evaluate exploration methods using multiple modalities and retrieval models in the context of multimedia. We apply the techniques in e-commerce product search and recommending, and demonstrate benefit for different retrieval scenarios. Lastly, we propose a method for extending database schema by latent visual attributes learned from image data. This enables closing the loop by going back to relational data, and potentially benefiting a range of industrial applications.