Real-time high-resolution hardware–software co-design neural architecture search for unmanned mobile platforms

Traditional manually designed high-resolution networks on mobile computing platforms often struggle to balance accuracy and inference speed. To address the issue of large computational costs in high-resolution neural networks, which makes them difficult to deploy on mobile computing platforms, we simplified the traditional multi-scale feature extraction process by reducing the three-branch fusion to a two-branch fusion, establishing a lightweight network-level search space. We applied gradient descent to iteratively optimize the two-layer parameters within the search space and used the pareto optimal algorithm to balance inference speed and accuracy. After convergence, we obtained a multi-scale feature extraction neural network structure that satisfies the balance inference speed and accuracy. When combined with different feature decoders, this structure enables real-time semantic segmentation and monocular depth estimation tasks on mobile platforms. An self-constructed unmanned mobile platform, built on a mobile computing platform, was used to collect image data from real-world environments to create a custom dataset. This dataset was used to validate the perception capabilities of the designed semantic segmentation and monocular depth estimation model on the mobile platform in real-world scenarios. The experiments demonstrate that our semantic segmentation model, designed for the NVIDIA NX mobile computing platform, achieves an accuracy of 71.7% for 1024 ×2048 high-resolution images, with an inference speed of 25.25 FPS. This represents a 39.2% improvement in inference speed over existing SOTA methods. Meanwhile, our monocular depth estimation model on the NVIDIA NX achieves an absolute relative error (Abs Rel) of 0.091, with an inference speed of 14.46 FPS. This method improves inference speed by 87.7% compared to existing methods, while preserving high accuracy. The code is available: https://github.com/douziwenhit/RealtimeSeg and https://github.com/douziwenhit/RealtimeMDE.