High-accuracy hyperspectro-polarimetric real-time imaging via a deep learning empowered infrared meta-sensor

Light inherently carries multidimensional information, including intensity, polarization, and spectrum. Employing a miniaturized device to simultaneously, instantaneously, and accurately capture the multidimensional information of incident light in a single exposure holds significant applications across numerous fields, but remains challenging. Here, we demonstrate high-accuracy hyperspectro-full-Stokes-polarimetric real-time imaging across a broad wavelength range (1150–1650 nm) with 167 spectral channels, via a metasurface integrated near infrared camera empowered by a residual attention network. The metasurface, which simultaneously performs polarization multiplexing and spectral dispersion, functions as a multidimensional encoder. A tailored deep learning network with a residual attention module is established and trained to reconstruct the multidimensional information of incident light with high accuracy. The key performance metrics—including spectral and image resolution, as well as spectral and polarization reconstruction accuracy—all surpass the best-reported specifications of hyperspectro-polarimetric cameras, with enhancements ranging from several-fold to one order of magnitude. Based on this approach, even intricately coupled multidimensional information can be resolved. In addition, the acquisition, processing, and inference time only takes 18 ms, allowing high-dimensional hyperspectral-polarimetric imaging at 55 frames per second in real time. This work offers a promising solution enabling high-accuracy hyperspectro-polarimetric real-time imaging with a compact camera.