Rectified Attention Gate Unit in Recurrent Neural Networks for Effective Attention Computation

Abstract

Recurrent Neural Networks (RNNs) have been successful in figuring out applications on time series data. Particularly, effectively capturing local features can ameliorate the performance of RNN. Accordingly, we propose a Rectified Attention Gate Unit (RAGU) which amends Gated Recurrent Unit (GRU) with two special attention mechanisms for RNNs. These two attention mechanisms are a Convolutional Attention (ConvAtt) module performing the convolutional operations on the current input and the previous hidden state to fairly establish the spatiotemporal relationship, and an Attention Module (AM) taking outputs from ConvAtt to fulfill the integrated attention computations for discovering the contextual dependency. Experimental results reveal that RNN using the proposed RAGUs has superior accuracies than RNNs using the other cell units on the HMDB51 and MNIST datasets. Therefore, RAGU proposed herein is an effective model which can bring out outstanding performance for various time series applications.