Infrared NeRF reconstruction based on perceptual pose and high-frequency-invariant attention

Abstract

Infrared images present significant challenges in novel view synthesis (NVS) due to low resolution and limited texture features. Additionally, the adaptive gain mechanism in infrared cameras leads to variations in the illumination across different viewpoints. These all can result in potential failures when utilizing infrared images for reconstructing Neural Radiance Fields (NeRF). To address these issues, we propose an end-to-end framework for pose estimation and rendering optimization. Specifically, perceptual pose optimization is used to estimate more accurate camera pose. To enhance the matching accuracy of multi scene corresponding points, we retain high-confidence camera poses while jointly optimizing both the scene and low-confidence poses. This allows for high-quality 3D scenes with accurate pose estimation for infrared images. The high-frequency-invariant attention module is designed to focus on the high-frequency features not easily captured and invariant edge information in infrared images by densely sampling, which can use high-frequency region to compensate for the low-frequency region differences caused by the adaptive gain mechanism. We evaluated our approach on datasets consisting of near-infrared, mid-wave infrared, and long-wave infrared images. Our method successfully reconstructs NeRF using infrared images and outperforms the state-of-the-art methods in terms of performance. The dataset is available at: https://github.com/YuangChen111/IR-NeRF