Biomimetic Visual System Implemented by a Two-Dimensional Plasmonic Photosensor Array for Processing Spatio-Temporal Information

Compared to machine vision chips, the human visual system can continuously perceive and process static scene and dynamic motion from the ever-changing real world with extremely low power consumption in both spatial and temporal dimensions. However, current neuromorphic devices either only focus on preprocessing and recognizing images in spatial dimension, or only focus on encoding and discriminating specific partial information (such as event-driven information), lacking biological fidelity. An in-sensor spiking neural network (SNN) is presented, consisting of a 2D plasmonic photosensor array (PPSA) that mimics the human visual system's ability to efficiently perceive, preprocess, encode, and process frame-driven spatio-temporal information. The photothermoelectric effect caused by plasmonic hot electrons enables the device to consume zero energy when perceiving full-pixel spatio-temporal data. Stochastic electrical signals with a 500 ns sampling period encode constant optical signals into spike trains, enabling the trained SNN to achieve 96% recognition accuracy and accurately interpret patterns with chaotic temporal order.