A method for automatically generating proper responses to user's utterances in open-ended conversation by retrieving documents on the web

We have been developing a new type of open-ended dialog system that generates a proper response to a user’s utterance using the abundant documents on the World Wide Web as sources. Existing knowledge-based dialog systems give meaningful information to users, but they are unsuitable for open-ended input. The system Eliza can handle open-ended input, but it gives no meaningful information. Our system lies between the above two dialog systems; it converses on various topics and gives meaningful information related to the user’s utterances. The system selects an appropriate sentence as a response from documents gathered through the Web, on the basis of surface cohesion ands shallow semantic coherence. We developed a trial system to converse about movies and experimentally found that the proposed method generated 66% appropriate responses.