More than Extracting "Important" Sentences: the Application of PEGASUS

Abstract

Pre-trained language models may reduce the amount of training data required. Among the models, PEGASUS, a recently proposed self-supervised approach, is trained to generate the pseudo-summary given the partially masked document. PEGASUS uses gap sentence generation for summarization. The most important sentences are masked, and then PEGASUS predicts the masked sentences as the output summary. In this study, however, we apply PEGASUS in a novel downstream task. We reformulate the task to generate the masked question part in a primary math word problem. In the past research, PEGASUS has shown good potentials on the few-shot datasets, so we try a smaller set of primary math text problems as well. The fine-tuning dataset sizes used in this study are 1000, 500, 50, 10 samples. Their performance are measured by a non-weighted average of the ROUGE-1, ROUGE-2, and ROUGE-L scores. The results show the outstanding performance of PEGASUS applied in our novel downstream task.