Assessing diffusion of spatial features in Deep Belief Networks

Abstract

Deep learning has recently gained popularity in many machine learning applications, but a theoretical grounding for the strengths, weaknesses, and implicit biases of various deep learning methods is still a work in progress. Here, we analyze the role of spatial locality in Deep Belief Networks (DBN) and show that spatially local information is lost through diffusion as the network becomes deeper. We then analyze an approach we developed previously, based on partitioning of Restricted Boltzmann Machines (RBMs), to demonstrate that our method is capable of retaining spatially local information when training DBNs. Specifically, we find that spatially local features are completely lost in DBNs trained using the “standard” RBM method, but are largely preserved using our partitioned training method. In addition, reconstruction accuracy of the model is improved using our Partitioned-RBM training method.