Prompt2Craft: Generating Functional Craft Assemblies With LLMs

Abstract

The Craft Assembly Task - a robotic assembly task inspired by handmade crafts - poses unique challenges relative to a traditional object assembly task.It involves open-ended decisions that are difficult to automate, with previous work relying on prior assumptions or expert knowledge, limiting scalability. In this work, we propose to address these limitations by employing an off-the-shelf Large Language Model (LLM) as the main decision maker in the available objects selection, pose estimation and connectivity between parts without additional training or fine-tuning, using the RGB image of the target object in the wild as a reference. The core of our method is a customized structure for the craft assembly description to constrain the LLM’s predictions. The proposals are validated through collision checks and physics simulation tests. We evaluate our approach on eight types of objects with three possible functions: “hit”, “support” and “rolling”. For visual similarity, we compare our generated final crafts with the 3D models generated by a state-of-the-art image-to-3D generation foundation model, and a baseline that uses image-to-3D part generation model. Our approach produces coherent, physically plausible assemblies, while the baseline often fails to generate or segment the object into parts suitable for assembly.