Generating a variety of expressions from visual information and user-designated viewpoints

Abstract

This paper reports the development and evaluation of a natural language generation system which generates a variety of language expressions from visual information taken by a CCD camera. The feature of this system is to generate a variety of language expressions from combinations of different syntactic structures and different sets of vocabulary, while managing the generation process based on the user-designated viewpoints. The system converts the visual information into a concept dependency structure using a semantic representation framework proposed by Takagi and Itoh. The system then transforms the structure and divides it into a set of words, deriving a word dependency structure, which is later arranged into a sentence. The transformation of a concept dependency structure and the variation in word segmentation allow the system to generate a variety of sentences from the same visual information. In this paper, we employ user-designated viewpoints to scenes containing more than one object. We designed the parameters of the user-designated viewpoints which enable the system to manage the generation process and to generate a variety of expressions. An evaluation has confirmed that the system generates certain variations according to parameter values set by the user. The variations include expressions referring to attribute values of the objects in the scenes and relative expressions denoting the relations between the targeted object and others.