A Model of Spoken Language Understanding Combining with Multi-Head Self-Attention

Abstract

Spoken Language Understanding (SLU) is a very important module in intelligent dialogue systems. It is usually constructed based on a bi-directional long and short-term memory network (BiLSTM). It has some shortcomings, such as relative single representation of feature space and fuzzy semantic features. For this reason, this study constructs a SLU model which combines the temporal characteristics of context and the characteristics of multi-layer representation space. The model combines a bi-directional long and short-term memory network and a multi-head self-attention to extract different feature information of contextual temporal features and multisemantic representation space of the text, respectively; then, the two features are fused using a residual linking method to enhance the features of word dependence at different locations within the text; meanwhile, the gate mechanism is then used to enable the intent detection task to establish an influence relationship on the slot filling task. Finally, the SNIPS dataset, the ATIS dataset, and the slot-gated model are selected for comparison experiments. The slot filling F1 value is increased by 4.14% and 1.1% respectively, and the accuracy of semantic framework is increased by 4.25% and 2.50% respectively. The results show the effectiveness of the model of SLU task.