Effective deep learning through bidirectional reading on masked language model

Abstract

Google BERT is a neural network that is good at natural language processing. It has two major strategies. One is “Masked language Model” to clear the word-level relationships, and the other is “Next Sentence Prediction” to clear sentence-level relationships. In the masked language model, with the task of masking some words in sentences, BERT learns to predict the original word from context. Some questions come to mind. Why BERT achieves effective learning by reading in two ways from fore and back? What is the difference between bidirectional reading? BERT learns to predict the original word using the surrounding words as context and to make two-way predictions by forward and backward readings in order to increase the precision. Besides, the bidirectional reading technique can be applied to scenario planning especially using back-casting from the future. This paper clarifies these mechanisms.