BERT-based Extractive Text Summarization of Scholarly Articles: A Novel Architecture

Abstract

Currently, there are a variety of extractive summarization approaches available, each with its own set of advantages and disadvantages. However, none of them are ideal, which means there is still room for advancement in this field of automation. BERT is a multilayer transformer network that has been pre-trained for a variety of self-supervised applications. However, because of its input length restriction, it is only appropriate for short text. As a result, we believe that using BERT for long document summarization will be a challenging task. We suggest a novel approach through which BERT can be utilized to summarize long documents. We used the method of dividing a whole document into multiple chunks and each chunk contains one sentence. The basic idea is to get sentence embeddings from BERT and then apply an encoder-decoder model on top of BERT. Experiments are conducted with two scholarly datasets (arXiv and PubMed). The results show that our technique consistently outperform several state-of-the-art models.