LLM-augmented hierarchical reinforcement learning for human-like decision-making of autonomous driving

Abstract

Reinforcement Learning (RL) has shown great promise for autonomous driving decision-making. However, such data-driven methods inherently struggle to be deployed in real-world due to their limited generalization to rare but safety-critical scenarios and low sample efficiency, resulting in high computational costs. To address these challenges, we propose a hierarchical RL framework augmented with a large language model (LLM), to enhance decision-making in complex driving environments through semantic understanding and commonsense knowledge. Inspired by human drivers, the LLM serves as an expert high-level planner that interprets textual descriptions of driving scenarios to generate a long-term goal point, a recommended meta-action, and a corresponding explanation, thereby navigating complex environments effectively. To meet real-time requirements, the high-level LLM module operates at a reduced frequency, balancing reasoning capability and inference latency. At the low level, however, it remains challenging for the RL agent to learn a sequence of continuous short-term actions, acceleration and steering, that can achieve the high-level goal while ensuring safety and efficiency. To bridge this gap, we introduce a Goal Gradient-based Transfer (GGT) mechanism that embeds an explicit gradient toward the LLM-generated goal, facilitating efficient policy learning. Additionally, to align the learned behaviors with human intents, we incorporate a human-in-the-loop reward design process.Specifically, the LLM contributes to reward design by generating structurally diverse functions, which are iteratively optimized using expert preferences over RL-generated trajectory pairs to ensure alignment with human values and safety. Overall, experimental comparisons in the CARLA simulator demonstrate that the proposed framework significantly improves generalization, interpretability, and human alignment in diverse and unseen driving scenarios.