Prefix-tuning-based Consistency Learning Framework for Machine Reading Comprehension with unanswerable questions

Machine reading comprehension (MRC) is a traditional yet challenging natural language processing task. With advancements in pre-trained language models, several approaches have achieved remarkable results. However, the widespread existence of unanswerable questions presents a new challenge to the MRC task. In particular, language models trained on a universal corpus commonly suffer from hallucinations, and tend to fabricate plausible answers to unanswerable questions. Moreover, most existing methods perform task-specific fine-tuning based on pre-trained language models, necessitating the storage and updating of a substantial number of parameters. To address these issues, we propose a lightweight framework called Prefix-tuning-based Consistency Learning Machine Reading Comprehension (PCMRC) to mitigate model hallucination and improve the accuracy of answering unanswerable questions. PCMRC is structurally composed of two similar answer extractors and a prefix-tuning-based unanswerable discriminator. The answer extractors extract both a correct answer and a plausible answer, while the unanswerable discriminator generates a global probability indicating whether the question is unanswerable. A consistency training mechanism is then introduced to balance the probability of unanswerability with the two extracted answers. In this way, the proposed PCMRC mitigates the overconfidence of pre-trained language models, reduces the interference of unanswerable questions by plausible answers, and ultimately improves the accuracy of handling unanswerable questions. Experiments on the SQuAD2.0 dataset show that PCMRC achieves an F1-score of 88.9% and an accuracy of 89.2% for unanswerable questions. Notably, this method requires fewer than 1% of the parameters compared to fine-tuning, demonstrating its superiority and efficiency in the MRC task.