Domain-Specific Contextualized Embedding: A Systematic Literature Review

Abstract

Word embedding has successfully resolved various Natural Language Processing (NLP) problems. Unfortunately, the method has a weakness in detecting polysemy and homonym. Those issues led to the emergence of a new approach, which is named contextualized embedding. Many researchers examined such embedding to resolve problems in various particular areas. However, the studies are published disparate and complex. To provide a more comprehensive overview of contextualized embedding research in specific domains, a Systematic Literature Review (SLR) was conducted. The SLR results show that research on domain-specific contextualized embedding pays more attention to solving NLP problems in the Healthcare domain, by the percentage of more than 65%, followed by the Academic & Research field and other areas. The popularity of the Healthcare domain is associated with the availability of abundant datasets, mostly in English. BERT is the most contextualized embedding models used for domain-specific tasks, followed by ELMo, and finally GPT-1, as well as XLNET. Almost all reviewed papers reported performance improvements by using domain-specific contextualized embedding in their proposed model. Contextualized embedding can resolve polysemy problems and reduces overfitting. Besides, many downstream tasks have proved the ease implementation of the embedding. The shortcomings of this embedding are the high requirements of computation resources, the long execution time, and the computation complexity. Domain-specific contextualized embedding has resolved many problems, mostly classification tasks (e.g., Question and Answering) and tagging tasks (e.g., Named Entity Recognition). The two evaluation methods for measuring the performance of domain-specific contextualized embedding are Intrinsic evaluation and Extrinsic evaluation.