Integrative AI framework for robotics: LLM-enabled reinforcement learning in object manipulation and task planning

Abstract

The paper develops an innovative hybrid AI framework that combines contextual reasoning of a large language model (LLM) with adaptivity of reinforcement learning (RL) for improved robotic object manipulation and task execution. In particular, the proposed system integrates high-level task planning, where GPT-4 and an RL submodule collaboratively generate optimized task strategies, with low-level real-time control through RL, allowing for enhanced adaptability in dynamic environments. The experimental results demonstrate significant improvements in task success rates and operational efficiency compared to standalone RL and GPT-4 approaches. In static environments, the integrative approach achieved a 90% task success rate, with an average completion time of 42.1 s and only 1.1 retries, outperforming RL-only (72%) and GPT-4-only (78%) methods. In dynamic environments, our integrative system maintained an 85% success rate, compared to 65% for RL-only and 70% for GPT-4-only. For complex tasks, the hybrid model showed a substantial advantage, with an 80% success rate, highlighting its superior performance in tasks requiring both high-level reasoning and low-level precision control.