Edge real-time tracking and FPGA-based hardware implementation for infrared tiny object

Infrared tiny object tracking holds significant application value in video surveillance and air defense systems. Deep convolutional neural networks (DCNNs) have demonstrated impressive performance in object tracking over the past few years. However, the high complexity and computing power requirements of the DCNN model make it difficult to deploy on power-sensitive and resource-constrained edge devices. To address this issue, we designed and implemented an FPGA-based infrared tiny object tracker by a hardware-software co-optimization approach, meeting the requirements for accuracy, latency, and power consumption under limited resources. First, we propose a lightweight and hardware-friendly object tracking network, SiamITO-Tiny, effectively improving the tracking accuracy for infrared tiny objects. Second, we design a full-mapping hardware acceleration architecture, mainly comprising a layer-fusion-based convolutional accelerator, a parallel pipelined adder tree, and an efficient data caching scheme. This architecture increases computational parallelism and data access bandwidth through layer fusion, loop optimization, pipelining, and array partitioning, while simultaneously balancing resource consumption and latency, thereby effectively improving computational performance and energy efficiency. Finally, the SiamITO-Tiny network is deployed on the Xilinx All Programmable SoC (ZYNQ) platform ZCU104. Experiments show that our method achieves 45.45 FPS and has a tracking score of 0.9. The computational performance reaches 307.2 GOP/s. The energy efficiency is 30.03 GOP/s/W, which is 39 and 6.81 times higher than it is on the CPU and GPU platforms, respectively. Compared to other acceleration methods, the energy efficiency is improved by 1.4 to 9.3 times, confirming the superiority of the proposed approach in edge real-time tracking.