High Spatio-Temporal Imaging Reconstruction for Hybrid Spike-RGB Cameras

Abstract

The acquisition of high-resolution image sequence for dynamic scenes of fast motion remains challenging due to motion blur caused by fast object movement. As a novel neuromorphic sensor, spike camera records the changing light intensity via spike stream of ultra-high temporal resolution, excelling in motion recording but limited in spatial resolution. This paper proposes a method for high spatio-temporal resolution (HSTR) imaging with a hybrid Spike-RGB camera, utilizing the information from spike stream to enhance the temporal resolution and the information from RGB images to enhance the spatial resolution of texture details. For this purpose, we present HSTR-Net, a dedicated network to process the spike and RGB data, which incorporates three key innovations: 1) A temporal control encoder enabling flexible temporal reconstruction through spike stream processing with embedded time parameters, eliminating the requirement to train multiple inference models; 2) Motion-aware feature projection that aligns RGB frame details to target timestamps using spike-derived motion offsets; 3) An adaptive transformer-based fusion strategy establishing cross-modal spatial correlations through mutual attention mechanisms. Extensive experiments demonstrate state-of-the-art performance on synthetic benchmark datasets with 5.23 dB PSNR and 6.94% SSIM improvement. It also shows visually impressive performance on real-world captured spike dataset.