Large language model assisted hierarchical reinforcement learning training

Traditional reinforcement learning (RL) cannot solve complex long-sequence decision tasks, especially when the environment rewards are sparse. Large language models (LLMs) can perform well in long-sequence decision tasks by leveraging their powerful inference capabilities. Although LLMs possess a large amount of general knowledge, LLM-based agents lack expertise in solving specific target problems. Considering that reinforcement learning models are smaller than LLMs and can be trained specifically to perform well on specific tasks, this paper proposes a hierarchical reinforcement learning framework assisted by a large language model, called LLMHRL. In this framework, the LLM acts as a teacher agent to guide the exploration of high-level policy in hierarchical reinforcement learning. The low-level policy consists of a library of selection-based policies. The agent executes specific actions based on the low-level policy chosen by the high-level policy. Furthermore, to reduce the action space of high-level policy, this paper decomposes it into skill options and target options. The two types of options are combined to obtain a high-level policy. This paper evaluates LLMHRL against baseline methods using both public and custom-built harder tasks across three environments: MiniGrid for key-door pairing, ManiSkill for tabletop sorting, and real-world scenarios. The results show that LLMHRL outperforms existing methods in success rate, convergence speed, and average return.