Comparison of Pretrained Models for Domain-Specific Entity Extraction from Student Report Documents

Abstract

The authors propose a methodology for extracting domain-specific entities (DSEs) from student report documents in the Russian language using pretrained transformer-based language models. Extracting DSEs from student report documents is a relevant task since the obtained data can be used for various purposes, ranging from the formation of project teams to the personalization of learning pathways. Additionally, automating the document processing workflow reduces the labor costs associated with manual processing. Expert-annotated student report documents are used as training material for training models. These documents were created by students in information technology programs between 2019 and 2022 for project-based and practical disciplines, as well as theses. The DSE extraction task is approached as two subtasks: named entity recognition (NER) and annotated text generation. A comparative analysis is conducted among NER encoder-only models (ruBERT, ruRoBERTa), encoder-decoder models (ruT5, mBART), and decoder-only models (ruGPT, T-lite) for text generation. The effectiveness of the models was evaluated using the F1-measure, together with an analysis of common errors. The highest F1-measure on the test set is achieved by mBART (93.55%). This model also shows the lowest error rate in DSE identification during text generation and annotation. The NER models demonstrated a lower tendency for errors but tended to extract DSEs in a fragmented manner. The obtained results indicate the applicability of the examined models for solving the stated tasks, considering the specific requirements of the problem.