$$\cal{Y}$$ -Tuning: an efficient tuning paradigm for large-scale pre-trained models via label representation learning

Abstract

With current success of large-scale pre-trained models (PTMs), how efficiently adapting PTMs to downstream tasks has attracted tremendous attention, especially for PTMs with billions of parameters. Previous work focuses on designing parameter-efficient tuning paradigms but needs to save and compute the gradient of the whole computational graph. In this paper, we propose \(\cal{Y}\)-Tuning, an efficient yet effective paradigm to adapt frozen large-scale PTMs to specific downstream tasks. \(\cal{Y}\)-Tuning learns dense representations for labels \(\cal{Y}\) defined in a given task and aligns them to fixed feature representation. Without computing the gradients of text encoder at training phrase, \(\cal{Y}\)-Tuning is not only parameter-efficient but also training-efficient. Experimental results show that for DeBERTa_XXL with 1.6 billion parameters, \(\cal{Y}\)-Tuning achieves performance more than 96% of full fine-tuning on GLUE Benchmark with only 2% tunable parameters and much fewer training costs.