Challenges and Opportunities of Using Transformer-Based Multi-Task Learning in NLP Through ML Lifecycle: A Position Paper

Abstract

The increasing adoption of natural language processing (NLP) models across industries has led to practitioners’ need for machine learning (ML) systems to handle these models efficiently, from training to serving them in production. However, training, deploying, and updating multiple models can be complex, costly, and time-consuming, mainly when using transformer-based pre-trained language models. Multi-Task Learning (MTL) has emerged as a promising approach to improve efficiency and performance through joint training, rather than training separate models. Motivated by this, we present an overview of MTL approaches in NLP, followed by an in-depth discussion of our position on opportunities they introduce to a set of challenges across various ML lifecycle phases including data engineering, model development, deployment, and monitoring. Our position emphasizes the role of transformer-based MTL approaches in streamlining these lifecycle phases, and we assert that our systematic analysis demonstrates how transformer-based MTL in NLP effectively integrates into ML lifecycle phases. Furthermore, we hypothesize that developing a model that combines MTL for periodic re-training, and continual learning for continual updates and new capabilities integration could be practical, although its viability and effectiveness still demand a substantial empirical investigation.