The Supervised Information Bottleneck.

Abstract

The Information Bottleneck (IB) framework offers a theoretically optimal approach to data modeling, although it is often intractable. Recent efforts have optimized supervised deep neural networks (DNNs) using a variational upper bound on the IB objective, leading to enhanced robustness to adversarial attacks. In these studies, supervision assumes a dual role: sometimes as a presumably constant and observed random variable and at other times as its variational approximation. This work proposes an extension to the IB framework and, consequent to the derivation of its variational bound, that resolves this duality. Applying the resulting bound as an objective for supervised DNNs induces empirical improvements and provides an information-theoretic motivation for decoder regularization.