Investigating a Hierarchical Inductive Bias in L2-textbook Seq2Seq Language Model

Abstract

The relations between words in natural language are governed by hierarchical structures rather than linear ordering. In recent years, artificial neural network-based language models (LMs) have demonstrated impressive achievements in tasks related to sentence processing. These models benefit from pre-training, which helps enhance their performance. However, our comprehension of the precise syntactic knowledge acquired by these models during sentence processing remains somewhat restricted. This paper examines whether the L2-textbook Seq2Seq (Sequence-to-Sequence) language model processes or transforms sentences based on a syntactic hierarchical inductive bias or a linear inductive bias through transformation tasks. We replicate several previous experiments and explore our model’s capacity to exhibit human-like behavior. Our experiments provide evidence that, in transformation tasks, our pre-trained L2-textbook LSTM-based Seq2Seq model performed based on the linear rule rather than hierarchical rule. In essence, our model showcased a linear inductive bias, consistent with the Scratch-Seq2Seq model.