Vision Powered Conversational AI for Easy Human Dialogue Systems

Abstract

In this paper, we propose an end to end goal-oriented conversational AI agent that can provide contextual information from a potential hazard site. We posit the conversational agent as a FloodBot capable of seeing, sensing, assessing hazard condition, and ultimately conversing about them. We present our domain-specific FloodBot design-solution and learning-experience from the real-time deployment in a flash flood devastated city that uses state-of-the-art deep learning models. We specifically used computer vision and pertinent natural language processing technologies to empower the conversation power of the FloodBot. To deliver such practical and usable AI, we chain multiple deep learning frameworks and create a human-friendly question-answer based dialogue system. We present our deployment details from the last five months and validate the results using ongoing COVID19’s impact on the area as well.