Work-in-Progress: Computing Sentence Similarity for Short Texts using Transformer models

Abstract

The field of natural language processing is being revolutionized with transformers. The latter is based on a novel type of neural network framework that is already pre-trained. Hence, large datasets to train models are no longer required. This framework is suitable for automated assessment systems (AAS), where a large number of labeled data is needed. The larger the dataset, the higher the accuracy of the AAS. In this work-in-progress paper, a prototype for an AAS has been built where two transformer models, namely the Sentence-Transformers from hugging face and the OpenAI GPT-3 models have been used. The transformer models generate the similarity index between students’ answers and reference answers from the Texas dataset. Then the similarity index is used to compute marks for students. The performance of the prototype is evaluated using the quadratic weighted kappa metric.