MTLV

Abstract

Multi-Task Learning (MTL) for text classification takes advantage of the data to train a single shared model with multiple task-specific layers on multiple related classification tasks to improve its generalization performance. We choose pre-trained language models (BERT-family) as the shared part of this architecture. Although they have achieved noticeable performance in different downstream NLP tasks, their performance in an MTL setting for the biomedical domain is not thoroughly investigated. In this work, we investigate the performance of BERT-family models in different MTL settings with Open-I (radiology reports) and OHSUMED (PubMed abstracts) datasets. We introduce the MTLV (Multi-Task Learning Visualizer) library for building Multi-task learning-related architectures which use existing infrastructure (e.g., Hugging Face Transformers and MLflow Tracking). Following previous work in computer vision, we clustered tasks and trained a separate model on each cluster (Grouped Multi-Task Learning (GMTL)). Contextual representation of the class labels (Tasks) and their descriptions was used by the library as features to cluster the tasks. We observed that grouping tasks for training with few models (GMTL) outperforms the MTL also GMTL is computationally more efficient than the STL setting (a separate model is trained for each task).