GNN Topology Representation Learning for Deformable Multi-Linear Objects Dual-Arm Robotic Manipulation

Deformable Multi-Linear Objects (DMLOs), or Branched Deformable Linear Objects (BDLOs), are flexible objects that possess a linear structure similar to DLOs but also feature branching or bifurcation points where the object’s path diverges into multiple sections. The representation of complex DMLOs, such as wiring harnesses, poses significant challenges in various applications, including robotic systems’ perception and manipulation planning. This paper proposes an approach to address the robust and efficient estimation of a topological representation for DMLOs leveraging a graph-based description of the scene obtained via graph neural networks. Starting from a binary mask of the scene, graph nodes are sampled along the objects’ estimated centerlines. Then, a data-driven pipeline is employed to learn the assignment of graph edges between nodes and to characterize the node’s type based on their local topology and orientation. Finally, by utilizing the learned information, a solver combines the predictions and generates a coherent representation of the objects in the scene. The approach is experimentally evaluated using a test set of complex real-world DMLOs. Within an offline evaluation, the proposed approach achieves a Dice score exceeding 90% in predicting graph edges. Similarly, the identification accuracy of branch and intersection points in the graph topology is above 90%. Additionally, the method demonstrates efficient performance, achieving a runtime of over 20 FPS. In an online assessment employing a dual-arm robotic setup, the approach is successfully applied to disentangle three automotive wiring harnesses, demonstrating the effectiveness of the proposed approach in a real-world scenario. Note to Practitioners—This research is motivated by the lack of effective robotic perception and manipulation solutions for deformable multi-linear objects such as wiring harnesses. These objects play a crucial role in several manufacturing industries. However, their intricate branching structures, inherent deformable nature, and limited distinctive visual characteristics present formidable challenges when automating tasks that involve them. The proposed approach addresses several of these challenges: 1) It maintains robustness in the presence of noise and disruptions in the input image mask due to its graph-based structure; 2) It remains unaffected by the existence of large connectors, clips, fixtures, and other objects along the branch sections of the object; 3) It avoids making any assumptions about the structure of the object; 4) It is capable of simultaneously identifying branch-points and resolving intersection-points among different branch sections of these objects. The experiments have verified the effectiveness of this solution when applied to real-world deformable multi-linear objects found in automotive settings. This underscores the potential to integrate this solution into industrial scenarios that involve the assembly and routing of such objects.