Multiple Words to Single Word Associations Using Masked Language Models

Abstract

In this paper, we examine a word association task that predicts a correct associative word from five stimulus words using Masked Language Models (hereafter referred to as MLMs). For MLMs, we used BERT and gMLP. Since our word association task uses only nouns for both stimulus and associative words, we trained new models by restricting masked tokens to nouns. In our experiment, we input sentences such as “The prefecture associated with Mt. Fuji, Lake Hamana, … and eels is MASK. (富士山、浜名湖、⋯、うなぎから連想する都道府県は MASK です。),” so that MASK outputs an associative word. In the experiments, we also examined adding Japanese quotation marks 「」 before and after the MASK, i.e., 「MASK」. The experiment results showed that the highest percentage of correct answers, 49%, was obtained by adding 「」 before and after the MASK (74% of the correct answers were within the top five words).