Learning hierarchical scene graph and contrastive learning for object goal navigation

Abstract

The task of object goal navigation (ObjNav) requires the agent to locate the given target object within a complex dynamic scene. To successfully accomplish the task, the agent needs to well understand the scenes, make executable decisions with less steps, avoid collisions, and successfully navigate to the target. As a result, efficient environmental perception and scene graph-inspired path planning is important to successfully accomplish the ObjNav task. In this paper, we present a hierarchical scene graph (HSG) contrastive learning, which consists of (1) a multimodal graph mixer that aligns the visual and textual information using open-vocabulary detector with GLIP. It can be regarded as an “eagle eye” to perceive target-related frontiers and suppress irrelevant information, and (2) a graph constructer that takes observed RGBD images to incrementally build a hierarchical scene graph. It acts as the “brain” that memorizes the common scene layout, (3) an action control contrastive learning that takes the graph contextual relationships as input to predict optimal actions to the target. It is treated as the “limbs” of the agent, coordinating and correcting incorrect movements. On the task of ObjNav, experiments on Gibson, HM3D, MP3D, and ProcTHOR demonstrate that navigation plans from the HSG framework achieve significantly higher success rates than existing map-based method, indicating the feasibility of executing navigation utilizing commonsense knowledge from language models leading efficient semantic exploration. Code is available at https://github.com/luosword/HSG4VN.