LLM based autonomous agent of human-robot collaboration for aerospace wire harnessing assembly

Abstract

The fusion of large language models (LLMs) and robotic system bring transformative potential to human-robot collaboration (HRC). Existing LLMs-based HRC methods mainly realize on fine-tune techniques, which has the shortcomings such as damage of the inherent ability of original LLMs, difficulty performing complex continuous task, less flexibility, fixed response strategy and computationally expensive. Alternatively, the development paradigm of LLM applications is transiting towards the autonomous agent mode. This paper proposed an interesting LLM agent based HRC framework (or HRC agent), which empowers the robot with human’s think mode and execution ability of sensing, interaction, self-reasoning, task planning and task execution. The chain-of-thought technique that generates a series of intermediate reasoning steps is adopted to improve the ability of LLMs to execute complex reasoning and task. Few-shot learning is used such that HRC agent can quickly learns new specific industry tasks by being provided a few examples. The reflection-based contextual memory mechanism enables HRC agent to have long term memory and continuous instruction understanding ability. A series of tools are developed and integrated into HRC agent, by which the capabilities of HRC agent can be easily expanded without much changing of the code framework. The functionality and effectiveness of HRC agent is validated in the aerospace wire harnessing assembly task, whose products has the characteristics of small diameter wires, complicated wire text, dense and tiny assembly holes, varying product batch size and customized production, and thus has high requirements for flexibility. The results show that the HRC agent is able to well understand the natural language instructions and give correct and effective response by chain-of-though, and subsequently, drive the robot to execute tasks correctly by calling tools.