Bidirectional Encoder Representation from Transformers (BERT) Variants for Procedural Long-Form Answer Extraction

Abstract

Extracting information from large verbose documents is a gruelling task which requires patience and huge amounts of effort. Lengthy documents like Portable Document Formats (PDFs) contain tonnes of information including tables, figures, etc. which makes it hard to retrieve specific pieces of textual content like step-by-step instructions or procedures. To the best of our knowledge, this work is the maiden effort to efficiently extract such long-form procedural answers from the aforementioned information sources. The proposed approach retrieves succinct responses from the relevant PDFs for a given user query using a transformer model embedded with attention mechanism. This model is trained using a self-made dataset namely, Pro-LongQA, consisting of carefully crafted procedure based questions and answers. A comparative study of one of the state-of-the-art transformer models, namely, Bidirectional Encoder Representation from Transformers (BERT) and its different variants such as RoBerta, Albert and DistilBert for the task of long-form question answering is performed. Among which, BERT and RoBerta proved to be the best performing models for this task with an accuracy of 87.2% and 86.4% respectively.