Real-time semantic SLAM-based 3D reconstruction robot for greenhouse vegetables

Abstract

Eggplant harvesting involves processes such as planting, monitoring, and spraying. However, in large eggplant plantations, relying solely on manual labor to complete these tasks not only requires extensive time and high capital costs but also results in extremely low efficiency. By combining unmanned vehicles and visual simultaneous localization and mapping (SLAM) for the three-dimensional (3D) reconstruction of an eggplant orchard we can intuitively obtain information regarding the orchard while providing a foundation for automating the aforementioned tasks. This study proposes a 3D semantic mapping and navigation solution for eggplant orchard based on visual SLAM. The baseline model of this system uses ORB-SLAM2, with improvements made by incorporating SCTNet-B, a semantic segmentation network, to enrich the representation of the point cloud map. In addition, in the tracking thread, a direct method is employed for feature extraction and matching, reducing computational load and enhancing real-time processing speed. Moreover, the point cloud map is converted into an OctoMap to reduce storage consumption. The system is also equipped with the A* algorithm and EGO-Planner algorithm for autonomous navigation of the unmanned vehicle. We implement the proposed system on an unmanned vehicle for field testing and evaluation. The experimental results demonstrate that the SCTNet-B achieves a mean intersection over union of 88.74% for semantic segmentation on a custom eggplant image dataset, with a frame rate of 28.36 FPS. After adopting the direct method, the system’s front-end tracking thread reduces the average processing time per frame by 55.68%. Using an OctoMap for storage of the semantic point cloud map results in an average reduction in memory consumption of 95.20%. The proposed system, integrated with the unmanned vehicle, meets the requirements for real-time semantic mapping and storage in large-scale eggplant orchard.