UAVO-NeRF: 3D reconstruction of orchards and semantic segmentation of fruit trees based on neural radiance field in UAV images

In precision agriculture, accurate 3D reconstruction of orchard environments is essential for crop health monitoring and automating agricultural tasks. This paper introduces UAVO-NeRF, a novel method using Unmanned Aerial Vehicles (UAVs) for high-fidelity 3D reconstruction and semantic segmentation of orchard scenes. To address inefficiencies in large-scale outdoor environments, we employ a nonlinear scene parameterization that compresses the unbounded scene into a cubic space, enabling denser sampling of distant points. We implement multi-resolution hash encoding to capture both global context and local details, significantly enhancing reconstruction speed and quality. To handle lighting variability, we incorporate appearance embeddings that adaptively encode lighting conditions, increasing the model’s robustness under diverse illumination. Our network’s output layer includes a 3D semantic segmentation module that distinguishes fruit trees from background elements, using a cross-entropy loss function to measure the difference between predicted and actual semantic labels. Depth prediction accuracy is improved using depth maps generated by a pre-trained monocular depth estimation model, refined through a composite loss function that combines reconstruction, depth, semantic, visibility, and interlevel losses to minimize artifacts and enhance geometric representation. Experimental results demonstrate that UAVO-NeRF achieves a Peak Signal-to-Noise Ratio (PSNR) of 23.82, outperforming state-of-the-art models like Instant-NGP and Mip-NeRF 360 across metrics such as PSNR, Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). Additionally, UAVO-NeRF achieves a mean Intersection over Union (mIoU) of 0.891 for fruit tree semantic segmentation from novel viewpoints, exceeding traditional 2D models by over 5%. This approach offers a robust technological solution for digital twin applications in agriculture.