Scene graph-driven reasoning for action planning of humanoid robot

Abstract

Recent progress in visual data analysis has significantly improved the ability of autonomous robots to understand their surroundings and perform complex tasks. This paper presents a modular method named Scene Graph-driven Reasoning for Action Planning (SG-RAPL) designed for high-level planning in dynamic environments, enabling adaptive control of humanoid robots. The method employs a three-dimensional (3D) scene graph to represent the environment and detect abnormal situations, while a large language model (LLM) translates natural-language commands into consecutive low-level actions. An original perceptual segmentation and tracking module constructs the scene graph in real time by providing instance segmentation, obstacle detection, and object pose estimation using data fusion with Augmented Reality University of Cordoba (ArUco) markers. The Planner module decomposes high-level tasks into subtasks such as navigation and object manipulation. Extensive experiments conducted on a manually collected and annotated dataset demonstrate that the proposed artificial intelligence-based approach efficiently plans complex actions in both virtual and real-world warehouse environments. The code and dataset of the proposed approach will be made publicly available.