Disentangled Representations by Pseudo-Maximum Mutual Information for Interpreting Multi-Layered Neural Networks

Abstract

The present paper aims to propose a new type of information-theoretic method to disentangle complex information to have easily interpretable representations for multi-layered neural networks. By this disentanglement of complex information, multi-layered neural networks can be easily compressed to the simplest ones with simple, linear and individual relations between inputs and outputs. The principal idea is to train neural networks by supposing maximum mutual information states before learning, namely, pseudo-maximum information maximization. This pseudo-maximum information method can greatly facilitate the implementation of maximum information procedures. The method was applied to the well-known Boston housing data set for easily reproducing the present results. The experimental results confirmed that pseudo-mutual information can be used to increase actual mutual information. In addition, when mutual information increased, compressed weights from multi-layered neural networks became similar to the correlation coefficients between inputs and targets of original data set. Thus, the method could successfully show that the main inference mechanism can be based on linear and individual relations between inputs and outputs with additional and peripheral nonlinear relations.