Perception-Based UAV Fruit Grasping Using Sub-Task Imitation Learning

Abstract

This work considers autonomous fruit picking using an aerial grasping robot by tightly integrating vision-based perception and control within a learning framework. The architecture employs a convolutional neural network (CNN) to encode images and vehicle state information. This encoding is passed into a sub-task classifier and associated reference waypoint generator. The classifier is trained to predict the current phase of the task being executed: Staging, Picking, or Reset. Based on the predicted phase, the waypoint generator predicts a set of obstacle-free 6-DOF waypoints, which serve as a reference trajectory for model-predictive control (MPC). By iteratively generating and following these trajectories, the aerial manipulator safely approaches a mock-up goal fruit and removes it from the tree. The proposed approach is validated in 29 flight tests, through a comparison to a conventional baseline approach, and an ablation study on its key features. Overall, the approach achieved comparable success rates to the conventional approach, while reaching the goal faster.