Patch Based Unsupervised Deep Learning for Retrieval and Part Matching in Large Image Datasets

Abstract

In this paper, we address the dual problems of retrieval and part (Region Of Interest (ROI)) matching, especially in large-scale image datasets. At the heart of our solution lies a patch-based unsupervised deep image representational model, termed a Bag of Variational Deep Embedded Visual Words (BoVDEVW). This model has three main components. Firstly, we extract a set of informative patches based on their entropy values. Secondly, the Variational AutoEncoder (VAE) is used to represent these informative patches in a robust low-dimensional probabilistic latent space. Third, deep embedded clustering is applied to group them. We introduce a composite loss function in this connection, which consists of reconstruction, regularization, and clustering losses, and explicitly show that it is convex. The cluster centers are termed as Variational Deep Embedded Visual Words. Image retrieval is performed by computing the Euclidean distance between a query image and the database images, each being represented as a Bag of Variational Deep Embedded Visual Words (BoVDEVW). The problem of finding ROI(s) in the retrieved images corresponding to ROI(s) in a query image is modeled as a matching problem in a bipartite graph constructed in the probabilistic latent space of the VAE. Comprehensive experimentation on large image datasets, NUS-WIDE and Google Landmarks Dataset (GLD) v2, and medium image datasets, Oxford 5K and Paris 6K clearly shows the efficacy of our proposed solution.