Intermediate-task transfer learning for Indonesian NLP tasks

Abstract

Transfer learning, a common technique in recent Natural Language Processing (NLP) research, involves pre-training a model on a large, unlabeled dataset using self-supervised methods and then fine-tuning it on a smaller, labeled dataset for a specific task. Recent studies have demonstrated that introducing an additional training step between pre-training and fine-tuning can further enhance model performance. This method is called intermediate-task transfer learning (ITTL). Although this approach can potentially improve performance in the target task, choosing an intermediate task that leads to the highest performance increase remains challenging. Furthermore, despite the extensive research on intermediate training methods in English NLP, the application of these techniques to Indonesian language processing is still relatively understudied. In this study, we apply the ITTL method to nine Indonesian NLP datasets, using each as both intermediate and target tasks, to investigate its behavior. Furthermore, we show that linear regression analysis can effectively identify factors that maximize performance improvement in target tasks when using ITTL. Our experiments reveal that ITTL enhances F1 score performance in the majority of cases, provided suitable intermediate tasks are selected. Specifically, our best performing model achieved performance gains in 9 out of 10 target task sets, with improvements reaching up to 18.6%. Our detailed analysis indicates that factors such as task type matching, task complexity, vocabulary size, and dataset size significantly influence the effectiveness of ITTL on target task performance. This research shows that ITTL, coupled with our proposed guidelines for intermediate task selection, offers a promising training paradigm for Indonesian NLP.