Chinese sentence based lexical similarity measure for artificial intelligence chatbot

Abstract

Artificial intelligence chatbots are computer programs that make interactions via auditory or textual information between human and machine using natural language processing techniques, most of which work on the basis of pattern matching. Typically, a chatbot recognizes the audio from human and translates it to text, and then matches the text with sentences stored in advance in the database by means of measuring similarity. So far, English chatbots have done much better than Chinese ones, mainly for the reason that processing sentences in Chinese requires more sophisticated techniques than that in English. Considering the low efficiency and accuracy of the existing Chinese sentence similarity measure methods, we propose a novel method oriented for lexical similarity measure of Chinese sentences by means of Chinese sentence segmentation and weighted word matching. Based on this method, a Chinese AI chatbot is developed and tested under a variety of settings. Experimental results show that our proposed method can achieve satisfactory performance in various cases.