Deep neural network compression via knowledge distillation for embedded applications

Abstract

Deep neural networks have shown significant success across various applications. To solve the complex problems, the increasing depth and complexity pose the challenges of large computation and storage requirements when deploying such networks on embedded devices with limited storage and power. Many techniques have been developed by researchers to compress the deep neural networks to make them deployable on portable devices by reducing the storage requirements. This paper describes the implementation of deep neural network with teacher student model. A comparatively smaller student model learns the information passed from the larger teacher model without losing the accuracy and its learning/inference rate is also improved. So this kind of framework is suitable for embedded applications deployment where real-time performance is required.