Short-Attention Mechanism for Generative Dialogue System

In recent years, generative dialogue has become the hottest topic in the field of Nature Language Process(NLP). Among the many suggested approaches, the Sequence-tosequence network framework, a variant of traditional Recurrent Neural Network(RNN), has attracted the attention of researchers because of its outstanding performance on many tasks. This model consists of an encoder which encoders the input sequence to a vector and a decoder that decodes the vector to the output sequence. Then attention was applied to the model, that is, the model assigns different weights to different parts to compute vector during decoding process. This end-to-end method enhances the ability to generate natural answers in the human-computer conversation process, while also increases its calculation costs. To solve the problem, we propose a novel short-attention mechanism, in which the original sequence is compressed to a shorter sequence before calculating weight and vector. We apply shortattention to dialogue systems tasks and the experimental results show that short-attention can shorten the computation time by about 20% compared to attention.