ET-PatchNet: A low-memory, efficient model for Multi-view Stereo with a case study on the 3D reconstruction of fruit tree branches

Abstract

The achievement of robotic fruit harvesting in intelligent farming depends heavily on the precise reconstruction of tree branch structures that direct harvesting arm movements. However, existing research in this field often grapples with computational inefficiencies and high costs. In response, we designed ET-PatchNet, a low-memory neural network for generating depth maps within the Multi-view Stereo (MVS) process, enabling efficient 3D reconstruction of branches. This highly Efficient network is based on Transformer and Patchmatchnet. ET-PatchNet incorporates an efficient backbone that includes self-attention and cross-attention mechanisms based on Transformer, which enriches global and 3D consistency information and enhances depth prediction accuracy and generalization performance. Furthermore, an adaptive depth resampling method has been developed, which is embedded in a iterative, coarse-to-fine, depth regression architecture based on learnable patches to minimize memory usage. To further amplify the representation capacity of depth characteristics, an auxiliary task has been integrated. Experimental results show that ET-PatchNet outperforms its competitors in completeness, computational efficiency, and low memory usage in evaluating the DTU and Tanks&Temples datasets. When predicting a single depth map at a resolution of 1152 × 864 pixels, it only took 0.13 $s$ to inference, with a memory usage of just $2824\mathrm{MB}$. Moreover, the 3D structure of observable branches on apple trees has been effectively reconstructed by fine-tuning our model on the BlendedMVS dataset. The mean and variance of distances between our reconstructed branch points and reference points are only $0.0292\mathrm{mm}$ and $0.0187{\mathrm{mm}}^2$. In conclusion, ET-PatchNet is ideal for integration into mobile embedded fruit harvesting equipment and exhibits significant potential for a wide range of applications.