A robotic grasping method of box-shaped objects based on Dual-Stream You Only Look Once framework

In the application area of sorting and placement, efficient sorting and precise placement of box-shaped objects have always been a key task, traditionally relying heavily on manual operation. With the advancement of industrial automation and artificial intelligence (AI), the application of AI in automation solutions offers a more effective approach to replacing manual labor, thereby enhancing production efficiency. At present, there is still a lack of box-shaped object classification and grasping methods for cluttered stacking scenes. To address this issue, this paper presents an AI-based approach that improves upon the You Only Look Once (YOLO) version 8 framework, enabling it to perform Red, Green, Blue, and Depth (RGBD) instance segmentation, called as Dual-Stream YOLO (DS-YOLO), and then combined with feature point matching algorithm, we can achieve precise recognition, grasping, and orderly placement of box-shaped objects in complex stacking environments. We created a synthetic dataset, referred to as Snack Box. Compared to state-of-the-art methods, the model trained with DS-YOLO on the Snack Box dataset achieves a 0.5% improvement in both mean average precision(mAP) at 50% intersection over union (IoU) and mAP from 50% to 90% IoU metrics, with a 93.3% grasp success rate. The average center point error is 6.20 millimeter, with plane normal vectors deviating by 4.27°and object angles by 6.26°on average. Additionally, our method outperforms state-of-the-art approaches on the Low-light Vision Visible-infrared Paired (LLVIP) dataset, with 15,488 aligned image pairs and pedestrian annotations. Our code and dataset are available at: https://github.com/DHR0703/YOLOv8_dual_Stream.