Log detection for autonomous forwarding using auto-annotated data from a real-time virtual environment

Object detectors for autonomous forestry operations have previously been developed mainly by training on physical manually annotated data, which is both time-consuming and costly. Since the ground truth in the virtual model is known, the training data can be auto-annotated, enabling the creation of larger training datasets, while also improving time and cost efficiency. In this work, a virtual environment in Unity is used in co-simulation with a real-time digital twin of a physical forestry vehicle, to generate realistic auto-annotated training data, as captured by an onboard stereo camera. First, it is shown that a log detector trained on physical data can detect logs in the virtual environment. Second, new detectors are trained, using different shares of virtual and physical data. It is shown that a detector trained using only virtual data, can learn to detect logs in the physical world. Moreover, virtual pre-training is shown to improve the performance of physically trained and tested detectors, both at low availability of physical training data, and in terms of domain generalization. A detailed detector performance analysis also highlights further potential and opportunities for future improvements. Furthermore, the real-time capable virtual models enable future machine learning tasks utilizing different levels of Hardware-in-the-Loop.