Towards visual-symbolic integration in the Soar cognitive architecture

Abstract

Computational models of visual reasoning are largely separate from models of non-visual reasoning and include only enough high-level reasoning to perform specific visual reasoning tasks, such as Raven’s progressive matrices or visual question answering. Although these models perform well at the pure visual reasoning tasks for which they are designed, their lack of a connection with a general-purpose high-level reasoning system means they cannot be applied to tasks requiring deliberate reasoning about both visual and non-visual knowledge. Simultaneously, many of the most mature and heavily studied cognitive architectures (e.g., Soar, ACT-R) feature only partial visual reasoning capabilities or none at all. This work describes initial efforts to create a visual reasoning system tightly integrated with a broader reasoning system by extending the Soar cognitive architecture with low-level visual memories and reasoning processes, and an evaluation of this system on tasks in a simple domain. Its ultimate aim is to demonstrate a path towards accommodating multiple levels of visual knowledge representations within an otherwise mostly symbolic, rules-based architecture.