Optimization of feature association strategies in multi-target tracking based on light field images

Abstract

Multi-object tracking (MOT) occupies an important position in the field of computer vision and has broad application prospects. One of the main challenges lies in extracting effective ReID features from complex scenes for data association. Complex scenes may cause targets to be detected but not tracked correctly. Light field cameras can simultaneously capture information from multiple viewing angles or light directions by using special optical components (such as microlens arrays) or sensor designs. Each pixel records the light entering the camera from different directions, allowing information from different viewing angles to be combined or reconstructed to form a complete image. To overcome this challenge, this paper utilizes light field cameras to acquire light field images and parse them into multi-view images. Through processing, the four-dimensional structure of the objects is mapped into epipolar plane images in specific directions. We designed an auto-encoder network to extract light field features from targets that were not successfully associated. In addition, we introduced a new strategy in the post-processing association stage of multi-target tracking by integrating the light field features into the general multi-target tracking framework as complementary features in the data association process. We have done a series of comparison and ablation experiments, and our multi-target tracking method achieves 86.4% of MOTA and 68.5% of HOTA in the same scenario, which is a better result compared to other SOTA tracking methods.