Deep Learning Method for Heliostat Instance Segmentation

Heliostat instance segmentation (HST-IS) is a crucial component of the heliostat tracking system at Heliogen’s Lancaster test facility. The system estimates the mirror normal of each heliostat by performing a nonlinear optimization-based fitting strategy using approximations of the non-shaded, non-blocked sunlit pixels on each heliostat, and the tracking system uses these estimates to improve performance. HST-IS is fundamentally challenging due to variability in lighting conditions and heliostat size relative to the capturing camera. Deep learning-based convolutional neural networks (CNN) have emerged in recent years by demonstrating noteworthy precision in tasks such as object recognition, detection, and segmentation. CNN-based methods offer a robust augmentation to HST-IS methods as they capture a context-less hierarchy of image features. In this study, we developed deep learning models to automatically segment heliostat instances from elevated images taken from the field. We study various image parameters and architectural customizations to optimize for scalability, robustness, and accuracy in our predictions. We perform robust evaluations of our best model to quantify gaps between model development and real-world deployment and provide evidence for utility in the field.