Beyond the Bias Variance Trade-Off: A Mutual Information Trade-Off in Deep Learning

Abstract

The classical bias variance trade-off cannot accurately explain how over-parameterized Deep Neural Networks (DNNs) avoid overfitting and achieve good generalization. To address the problem, we alternatively derive a Mutual Information (MI) trade-off based on the recently proposed MI explanation for generalization. In addition, we propose a probabilistic representation of DNNs for accurately estimating the MI. Compared to the classical bias variance trade-off, the MI trade-off not only accurately measures the generalization of over-parameterized DNNs but also formulates the relation between DNN architecture and generalization.