Visual Imitation Learning for robotic fresh mushroom harvesting

Abstract

Imitation Learning holds significant promise in enabling the automation of complex robotic manipulations tasks which are impossible to explicitly program. Mushroom harvesting is a task of high difficulty requiring weeks of intense training even for humans to master. In this work we present an end-to-end Imitation Learning pipeline that learns to apply the series of motions, namely reaching, grasping, twisting, and pulling the mushroom directly from pixel-level information. Mushroom harvesting experiments are carried out within a simulated environment that models the mushroom dynamics based on von Mises yielding theory with parameters obtained through expert picker demonstration wearing gloves with force sensors. We test the robustness of our technique by performing randomization on the camera extrinsic and intrinsic parameters as well as on the mushroom sizes. We also evaluate on different kinds of visual input namely grayscale and depth maps. Overall, our technique shows significant promise in automating mushroom harvesting directly from visual input while being remarkably lean in terms of computation intensity. Our models can be trained on a standard Laptop GPU in under one hour while inference of an action takes less than 1.5ms on a Laptop CPU. A brief overview of our experiments in video format is available at: https://bit.ly/41kCH7T