DisT5: A Text-to-Text transformer model for disaster events

Abstract

Transformer-based language models have become essential components in constructing pipelines for various NLP tasks. Although various pre-trained transformer models have been developed for different tasks, they still fail to achieve the desired performance in the disaster domain. Consequently, a lack of pre-trained models specifically designed for this domain. Therefore, in this paper, we present DisT5, a pre-trained transformer model specific to the disaster domain. We have adopted a T5-style self-supervised pre-training approach. We further pre-trained the T5-model on a large collection of textual data covering diverse disaster events, both natural and man-made. We benchmark DisT5 on three downstream tasks: (1) Tweet category classification, (2) Key-phrase identification, and (3) Abstractive summarization. We validate the performance of DisT5 for the aforementioned tasks through comprehensive experiments. Our experimental results demonstrate that additional pre-training improves the performance of the DisT5 model in all the aforementioned tasks. We achieved up to 70.35% accuracy improvement in tweet classification, 87.65% IOU (F1-score) improvement in key-phrase identification, and up to 76.47% and 72.72% ROUGE-N F1-score improvement in summarization without and with fine-tuning, respectively. The pre-trained model will be made available at https://huggingface.co/Piyyussh/DisT5.