Robotic manipulation framework based on semantic keypoints for packing shoes of different sizes, shapes, and softness

With the rapid development of the warehousing and logistics industries, the packing of goods has gradually attracted the attention of academia and industry. The packing of footwear products is a typical representative paired-item packing task involving irregular shapes and deformable objects. Although studies on shoe packing have been conducted, different initial states due to the irregular shapes of shoes and standard packing placement poses have not been considered. This study proposes a robotic manipulation framework, including a perception module, reorientation planners, and a packing planner, that can complete the packing of pairs of shoes in any initial state. First, to adapt to the large intraclass variations due to the states, shapes, and deformation of shoes, we propose a vision module based on semantic keypoints, which can also infer additional information such as sizes, states, poses, and manipulation points by combining geometric features. Subsequently, we not only propose primitive-based reorientation methods for different states of a single deformable shoe but also propose a fast reorientation method for the top state using box edge contact and gravity, which further improve the efficiency of reorientation. Finally, based on the perception module and reorientation methods, we propose a task planner for packing paired shoes in any initial state to provide an optimal packing strategy. Real-world experiments were conducted to verify the robustness of the reorientation methods and the effectiveness of the packing strategy for various types of shoes. In this study, we highlight the potential of semantic keypoint representation, introduce new perspectives on the reorientation of 3D deformable objects and multi-object manipulation, and provide a reference for paired object packing.