ISP-free multi-spectrum fused imaging for extremely low-light enhanced photography

Abstract

Achieving high-quality imaging under extremely low-light conditions is crucial for autonomous driving and night surveillance applications. Traditional approaches predominantly focus on the post-processing of degraded RGB data, which struggle to effectively mitigate noise in very low-light situations with limited input information and significant noise interference. In this study, a computational multi-spectral fusion imaging framework is proposed to enhance low-light images by encoding a broader spectrum of optical source information into the imaging pipeline. An end-to-end spectral fusion network (SPFNet), consisting of an encoder for the automatic extraction of scene features and decoder for channel fusion, is designed to integrate spectral fusion with image denoising. Utilizing the novel Multi_Conv module, a diverse range of spectral features are extracted from multi-spectral raw data, providing multi-scale cross-references for noise suppression, thereby facilitating high-quality image fusion. A pilot optical system was built to capture a real-scene multi-spectral-RGB dataset under illuminance conditions below 0.01 lx per spectrum. Experimental results confirm that the proposed method significantly outperforms traditional RGB imaging techniques, demonstrating an average improvement of over 7.87 dB in peak signal-to-noise ratio (PSNR) and 0.25 in structural similarity index (SSIM). Comprehensive ablation and contrast experiments were conducted to verify that the proposed model achieved the best performance in terms of detail reconstruction and color fidelity. Eschewing the need for a cumbersome traditional image signal processing (ISP) pipeline and strict experimental constraints, the proposed framework offers a novel and viable solution for extreme low-light imaging applications, including portable photography, space exploration, remote sensing, and deep-sea exploration.