Espresso: Exploiting the Sparsity Property in Brain-Inspired Vision Sensors With Spatiotemporal Ordering

Abstract

Brain-inspired vision sensors (BVSs), drawing inspiration from the human visual system, produce sparse, high-temporal-resolution data stream capable of capturing rapid object motion. However, real-time processing of such data while preserving its inherent sparsity presents a critical challenge for practical deployment. A key challenge is to leverage the spatiotemporal correlations among event stream (ES). Overlooking these correlations leads to prohibitively high query costs that even negate the benefits of sparsity while exploiting them introduces complications such as input–output order conflicts and tradeoffs between memory usage and latency. To address this, we present Espresso, an efficient hardware architecture that leverages the spatiotemporal order of events while explicitly preserving sparsity, achieving low-latency stream processing of event data. We first formalize a spatiotemporal order representation that identifies key features for stream processing on sparse events. Building on this, Espresso decouples the output window address from the input event address, resolving order conflicts via a dedicated queue mechanism and minimizing memory overhead through an optimized hash table. This enables immediate window-wise processing with minimal latency. To coordinate the pipeline, we design the Event-Scheduler, a streamlined finite state machine (FSM) that prunes computations on zero values and aligns input–output stream order discrepancies. Integrated together, these modules deliver scalable, high-throughput processing for event-driven vision tasks. Espresso achieves up to 5000 frames/s, offering a $5.1\times $ performance improvement over embedded GPUs. With parallel instantiations, it exceeds over 7000 frames/s in structured scenes and maintains over 2000 frames/s under complex-environment scenarios with minimal hardware overhead. These results establish Espresso as an efficient and scalable solution for real-time event-based vision processing, demonstrating the importance of spatiotemporal ordering in unlocking the full potential of BVSs.