A neuromorphic photodetector with ferroelectric-controlled static, event, and short-term memory modes for on-chip real-time spatiotemporal classification and motion prediction

Event-based vision sensors offer sparse, low-latency alternatives to frame-based imaging, but their lack of embedded memory and static scene awareness limits use in intelligent systems. Most designs capture only transient changes and rely on external processors for classification and motion prediction, lacking the temporal continuity and energy efficiency needed for real-time, context-aware decision-making. Here, we report a neuromorphic photodetector that integrates voltage-controlled static sensing, event detection, and tunable short-term memory (STM) within a single pixel. By combining a photoactive silicon layer with a ferroelectric HfZrO₂ stack, the device enables bias-dependent transitions between self-powered event spikes (∼73 µs), steady-state photocurrent, and programmable STM-like decay responses. By coupling the sensor array to a field-programmable gate array that performs on-chip learning and inference, the system intrinsically encodes real-time temporal dynamics to directly classify spatiotemporal patterns—such as gestures, logic sequences, and Morse code—with over 93 % accuracy, while also enabling real-time motion prediction of dynamic objects. The resulting architecture reduces power consumption by over 1000 × and boosts inference speed by more than 200 × compared to conventional event sensors with software-based neural networks, while STM elevates prediction accuracy from 20 % to over 80 % in dynamic position tracking tasks. This unified sensor–processor platform offers a scalable route toward compact, adaptive, and low-power neuromorphic vision systems.