Sparse Neural Networks with Topologies Inspired by Butterfly Structures

Abstract

In this paper we propose sparse neural networks with topologies inspired by fast computational butterfly structures of selected linear transforms, namely: fast discrete cosine transform, and Beneš network like topologies. We demonstrate that sparse neural networks allow to obtain high reduction in the number of arithmetic operations and weights needed by neural structures while preserving good efficiency in selected tasks where dense neural networks find their applications. In order to verify the efficiency of the considered structures we conducted a series of experiments in data compression and image recognition. The obtained experimental results confirm good efficiency of the considered sparse neural structures and reveal that such topologies allow for significant reduction in the number of arithmetic operations and weights that must be trained and stored in order to re-use trained neural networks.