Pivot-based Unsupervised Domain Adaptation for Pre-trained Language Model

Abstract

In the task of text classification, natural language processing technology provides an effective solution for automatically identifying text content classification, but labeled data is difficult to obtain in specific domains. To reduce manual labeling, some researchers have proposed unsupervised domain adaptation technology, which is a special transfer learning technology, transferring source domain models suitable for general knowledge to the target domain with less labeled data, to improve the generalization effect of the model in the target domain. However, the current unsupervised domain adaptation methods are mostly to fine-tune the pre-trained model directly using unsupervised data from the target domain. This method needs a large amount of unsupervised data as usual to improve the training effect. Therefore, this paper presents a pivot-based unsupervised domain adaptation method, which extracts and masks pivots from unsupervised data, fine-tunes the pre-trained language model, and finally validates the method using supervised training, compared with the method of directly using unsupervised data to fine-tune the original model. The pivot-based domain adaptation method effectively improves the efficiency of domain knowledge transfer for the specific domain.