BinaryEye: A 20 kfps Streaming Camera System on FPGA with Real-Time On-Device Image Recognition Using Binary Neural Networks

Abstract

Streaming high-speed cameras pose a major challenge to distributed cyber-physical and IoT systems, because large data volumes need to be transferred under stringent realtime constraints. Edge processing can mitigate the data deluge by extracting relevant information from image data on-device with low latency. This work presents an FPGA-based 20 kfps streaming camera system, which can classify regions of interest (ROI) within a frame with a binarized neural network (BNN) in realtime streaming mode, achieving massive data reduction. BNNs have the potential to enable energy-efficient image classifications for on-device processing. We demonstrate our system in a case study with a simple real-time BNN classifier achieving 19.28 us latency at 0.52 W power consumption and resulting in a 980x data reduction. We compare external image processing with this result, showing 3x energy savings, and discuss the used HDL/HLS design flow for BNN implementation.