The Role of RNNs for Contextual Representations: A Case Study Using DMN-plus

Recurrent neural networks (RNNs) have been used prevalently to capture long-term dependencies of sequential inputs. In particular, for question answering systems, variants of RNNs, such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), allow the positional, ordering or contextual information to be encoded into latent contextual representations. While applying RNNs for encoding this information is intuitively reasonable, no specific research has been conducted to investigate how effective is their use in such systems when the sequence of sentences is unimportant. In this paper we conduct a case study on the effectiveness of using RNNs to generate context representations using the DMN+ network. Our results based on a three-fact task in the bAbI dataset show that sequences of facts in the training dataset influence the predictive performance of the trained system. We propose two methods to resolve this problem, one is data augmentation and the other is the optimization of the DMN+ structure by replacing the GRU in the episodic memory module with a non-recurrent operation. The experimental results demonstrate that our proposed solutions can resolve the problem effectively.