Tasks-Embedded Reparameterization: A Novel Framework for Task-Specific Transfer Enhancement With Multitask Prompt Learning

Abstract

Current​ fine-tuning techniques for large pretrained language models (LLMs) face significant challenges, particularly regarding the high computational costs associated with adapting billions of parameters and their limitations in effectively addressing diverse language understanding tasks. These methods often result in an inability to manage inter-task dependencies effectively, leading to underutilization of inter-task information. To address these issues, we propose tasks-embedded reparameterization (TER), a novel parameter-efficient fine-tuning framework that exploits multitask learning to enhance task-specific capabilities. The TER model integrates prompt tuning and multitask reparameterization, merging task-specific experts and hidden states of target tasks in a unified model framework. Furthermore, it employs a dynamic, task-oriented gating mechanism to optimize the prompts output by the model. This method dynamically adjusts the parameters according to the differing requirements of the task, ensuring that the model optimally adjusts the parameters according to the specific requirements of the task, so that the task can find a suitable balance between different tasks and improve knowledge sharing and task adaptability. Experimental evaluations using the SuperGLUE benchmark demonstrate that TER consistently outperforms existing parameter-efficient fine-tuning techniques in both performance and computational efficiency, offering a promising solution for task-specific language understanding in both research and industry.