A Novel Attention-Based Layer Pruning Approach for Low-Complexity Convolutional Neural Networks

Abstract

Deep learning (DL) has been very successful for classifying images, detecting targets, and segmenting regions in high-resolution images such as whole slide histopathology images. However, analysis of such high-resolution images requires very high DL complexity. Several AI optimization techniques have been recently proposed that aim at reducing the complexity of deep neural networks and hence expedite their execution and eventually allow the use of low-power, low-cost computing devices with limited computation and memory resources. These methods include parameter pruning and sharing, quantization, knowledge distillation, low-rank approximation, and resource efficient architectures. Rather than pruning network structures including filters, layers, and blocks of layers based on a manual selection of a significance metric such as l1-norm and l2-norm of the filter kernels, novel highly efficient AI-driven DL optimization algorithms using variations of the squeeze and excitation in order to prune filters and layers of deep models such as VGG-16 as well as eliminate filters and blocks of residual networks such as ResNet-56 are introduced. The proposed techniques achieve significantly higher reduction in the number of learning parameters, the number of floating point operations, and memory space as compared to the-state-of-the-art methods.