Bridging the sim2real gap: Training deep neural networks for heliostat detection with purely synthetic data

Deep neural networks have demonstrated remarkable success in image processing across various domains. However, to achieve state-of-the-art performance, a substantial amount of high-quality training data is essential. In the context of optical heliostat monitoring, acquiring such data remains a challenge which is why deep neural networks are still scarcely used. We propose the use of synthetic training data to address this deficit and conduct a comprehensive investigation of scene parameters within our simulation environment to mitigate the sim2real gap. Our findings demonstrate that training models for object and keypoint detection in aerial images of heliostat fields with purely synthetic data is feasible and yields promising results with the appropriate scene configuration. Our best model achieves an average precision (AP) of 0.63 in heliostat detection and accurately detects 61% of outer mirror corners on our test dataset, comprising six manually annotated real-world drone images of a heliostat field. By evaluating the model on a simulated replication of this test dataset, we measure a remaining sim2real gap of 30% and 35% for the respective tasks. Furthermore, we showcase the model’s transferability to other heliostat geometries. By generating an additional 200 synthetic images showing the new geometry and performing a brief fine-tuning of the model, we achieve promising qualitative results on real-world images of another plant. To the best of our knowledge, this work is the first application of deep learning achieving such results in mirror corner detection in airborne imagery of heliostat fields while offering a straightforward approach for power plant transfer.