Learning and grounding visual multimodal adaptive graph for visual navigation

Visual navigation requires the agent reasonably perceives the environment and effectively navigates to the given target. In this task, we present a Multimodal Adaptive Graph (MAG) for learning and grounding the visual clues based on the object relationships. MAG consists of key navigation elements: object relative position relationships, previous navigation actions, past training experience, and target objects. This enables the agent to accurately gather multimodal information and find the target faster. Technically, our framework performs continuous modeling of pre-trained vision–language grounding model to align the multimodal graph, text information with visual perception. For output, we introduce constraints on the graph’s value estimation (GVE) functions to supervise the agent predict optimal actions, which can help it escape from deadlocks. With the MAG, the agent can effectively perceive the environment and get optimal actions. We train our framework with human demonstration and collision signals. Results demonstrate that our approach improves by 10.1% in SPL (Success weighted by Path Length) and 25.4% in success rate relative to the baseline method in the AI2THOR environment. Our code will be publicly released in the scientific community.