Local Manifold Regularization for Knowledge Transfer in Convolutional Neural Networks

In this work we present a method for local manifold-based regularization, as a mechanism for knowledge transfer during training of Convolutional Neural Networks. The proposed method aims at regularizing local features produced in intermediate layers of a “student” CNN through an appropriate loss function that encourages the model to adapt such that the local features to exhibit similar geometrical characteristics to those of an “instructor” model, at corresponding layers. To that purpose we formulate a computationally efficient function, loosely encoding the neighboring information in the feature space of the involved feature sets. Experimental evaluation demonstrates the effectiveness of the proposed scheme under various scenarios involving knowledge-transfer, even for difficult tasks where it proves more efficient than the established technique of knowledge distillation. We demonstrate that the presented regularization scheme, utilized in combination with distillation improves the performance of both techniques in most tested configurations. Furthermore, experiments on training with limited data, demonstrate that a combined regularization scheme can achieve the same generalization as an un-regularized training with 50% of the data.