Segmentation of surface and subsurface damages in concrete structures through fusion of multi-modal images using vision transformer

Semantic segmentation of multimodal images combining visible and infrared spectra enables quantification of both surface and subsurface damage in concrete structures. High-quality segmentation, however, hinges on precise cross-modal registration and an effective fusion strategy. Sparse feature similarity across these modalities typically observed in real-world infrastructure images, limits the effectiveness and generalizability of existing registration algorithms. To overcome this limitation, this paper proposes a new multi-modal image registration algorithm that narrows the search space leveraging epipolar constraint and employs a modified multi-scale mutual-information metric for robust feature matching. Tests on a purpose-built dataset show the method surpasses state-of-the-art registration algorithms. The paper also evaluates how fusion schemes and loss functions affect segmentation performance, revealing that a combined loss function (i.e., OHEM cross entropy and Generalized Dice Loss) paired with an early-fusion strategy yields the highest mean Intersection-over-Union. These contributions advance a comprehensive framework for automated damage segmentation in multimodal imagery.