Improved Feed Forward Attention Mechanism in Bidirectional Recurrent Neural Networks for Robust Sequence Classification

Abstract

Feed Forward Attention (FFA) in Recurrent Neural Networks (RNNs) is a popular attention mechanism to classify sequential data. In Bidirectional RNNs (BiRNNs), FFA concatenates hidden states from forward and backward layers to compute unscaled logits and normalized attention weights at each time step and softmax is applied to the weighted sum of logits to compute posterior probabilities. Such concatenation corresponds to the addition of individual unnormalized attention weights and unscaled logits from forward and backward layers. In this paper, we present a novel attention mechanism called the Improved Feed Forward Attention Mechanism (IFFA), that computes the probabilities and normalized attention weights separately for forward and backward layers without concatenating the hidden states. Finally, weighted probabilities are computed at each time step and averaged across time. Our experimental results show IFFA outperforming FFA in diverse classification tasks such as speech accent, emotion and whisper classification.