Towards robotic deep spatiotemporal language understanding based on mental-image-directed semantic theory

Among subsets of natural language, spatial language, more exactly spatiotemporal language here, has been considered most essential for human-like interaction between people and robots expected in near future. Quite distinctively from conventional learning-based approaches to natural language understanding (NLU), mental-image-directed theory (MIDST) proposes a robotic deep NLU methodology based on a mental image model as a formal system for knowledge representation and reasoning. The application system named CoMaS is designed to understand User’s utterances in text and respond in text or animation through human-like spatiotemporal reasoning based on the mental image model. In this work, CoMaS was compared with human subjects through a psychological experiment on spatiotemporal language understanding and showed globally good agreement with them and locally some interesting and reasonable disagreement. This kind of disagreement was found among the human participants as well and explainable as difference in personal conceptualization or reasoning based on mental image. The experimental results and theoretical discussion based on them showed well the effectiveness and uniqueness of our study.