Joint Reconstruction and Spatial Super-Resolution of Hyper-Spectral CTIS Images via Multi-Scale Refinement

Abstract

The Computed Tomography Imaging Spectrometer (CTIS) is a snapshot imaging device that captures Hyper-Spectral images as two-dimensional compressed sensor measurements. Computational post-processing algorithms are later needed to recover the latent object cube. However, iterative algorithms typically used to solve this task require large computational resources and, furthermore, these approaches are very sensitive to the presumed system and noise models. In addition, the poor spatial resolution of the $0$ th diffraction order image limits the usability of CTIS in favor of other snapshot spectrometers even though it enables higher spectral resolution. In this paper we introduce a learning-based computational model exploiting a reconstruction network with iterative refinement, that is able to recover high quality hyper-spectral images leveraging complementary spatio-spectral information scattered across the CTIS sensor image. We showcase the reconstruction capability of such model beyond the spatial resolution limit of the $0$ th diffraction order image. Experimental results are shown both on synthetic data and on real datasets that we acquired using two different CTIS systems coupled with high spatial resolution ground truth hyper-spectral images. Furthermore, we introduce HSIRS, the largest dataset of its kind for joint spectral image reconstruction and semantic segmentation of food items with high quality manually annotated segmentation maps and we showcase how hyper-spectral data allows to efficiently tackle this task.