Learn Fine-grained Sharing Network for Multiple Tasks

Conventional Multi-Task Learning (MTL) models, such as hard sharing, adopt handcrafted network architecture, which shares entire layers for all tasks, and thus have two shortcomings: 1) negative transfer phenomenon and 2) low parameter efficiency. This paper proposes a novel neural network model, which allows different tasks to share a network at the parameter level. Specifically, the model defines a subnet for each task by adopting task-specific binary masks. The masks are trainable and can be learned together with network weights using standard back-propagation. Benefit from the fine-grained sharing mechanism, the negative transfer phenomenon can be alleviated, and the parameter efficiency is greatly improved. According to the experiments on a public dataset, our model outperforms the single-task baseline model even when only 0.8% of parameters remained in the subnets. Compared with the multi-task baseline model using fixed masks, our model is much more robust to changes in network sparsity.