Transfer Learning with Shapeshift Adapter: A Parameter-Efficient Module for Deep Learning Model

Abstract

Fine-tuning pre-trained models is arguably one of the most significant approaches in transfer learning. Recent studies focus on methods whose performance is superior to standard fine-tuning methods, such as Adaptive Filter Fine-tuning and Fine-tuning last-k. The SpotTune model outperforms most common fine-tuning methods due to a novel adaptive fine-tuning approach. Since there is a trade-off between the number of parameters and performance, the SpotTune model is not parameter efficient. In this paper, we propose a shapeshift adapter module that can help reduce training parameters in deep learning models while pre-serving the high-performance merit of SpotTune. The shapeshift adapter yields a flexible structure, which allows us to find a balance between the number of parameters and performance. We integrate our proposed module with the residual blocks in ResNet and conduct several experiments on the SpotTune model. On the Visual Decathlon Challenge, our proposed method gets a score close to SpotTune and it outperforms the SpotTune model over half of the datasets. Particularly, our proposed method notably uses only about 20% of the parameters that are needed when training using a standard fine-tuning approach.