Double-Feedback: Enhancing Large Language Models Reasoning in Robotic Tasks by Knowledge Graphs

Abstract

Large language models (LLMs) have demonstrated remarkable reasoning capabilities. However, in real-world robotic tasks, LLMs face grounding issues and lack precise feedback, resulting in the generated solutions deviating from the actual situation. In this letter, we propose Double-Feedback, a method that enhances LLMs reasoning by Knowledge graphs (KGs). The KGs play three key roles in Double-Feedback: prompting the LLMs to generate solutions, representing the task scenes, and verifying the solutions to provide feedback. We design structured knowledge prompts that convey the task knowledge background, example solutions, revision principles, and robotic tasks to the LLMs. We also introduce the distributed representation to quantify the task scene with interpretability. Based on the structured knowledge prompts and the distributed representation, we employ the KGs to evaluate the feasibility of each step before execution and verify the effects of the solutions after completing the tasks. The LLMs can adjust and replan the solutions based on the feedback from the KGs. Extensive experiments demonstrate that Double-Feedback outperforms prior works in the ALFRED benchmark. In addition, ablation studies show that Double-Feedback guides LLMs in generating solutions aligned with robotic tasks in the real world.