Simultaneous Sparsity and Parameter Tying for Deep Learning Using Ordered Weighted ℓ1 Regularization

A deep neural network (DNN) usually contains millions of parameters, making both storage and computation extremely expensive. Although this high capacity allows DNNs to learn sophisticated mappings, it also makes them prone to over-fitting. To tackle this issue, we adopt a recently proposed sparsity-inducing regularizer called OWL (ordered weighted ℓ1, which has proven effective in sparse linear regression with strongly correlated covariates. Unlike the conventional sparsity-inducing regularizers, OWL simultaneously eliminates unimportant variables by setting their weights to zero, while also explicitly identifying correlated groups of variables by tying the corresponding weights to a common value. We evaluate the OWL regularizer on several deep learning benchmarks, showing that it can dramatically compress the network with slight or even no loss on generalization accuracy.