LVIMOT: Accurate and robust LiDAR-visual-inertial localization and multi-object tracking in dynamic environments via tightly coupled integration

Most existing LiDAR-visual-inertial (LVI) localization systems typically assume a static environment and often neglect the valuable dynamic object information, which limits localization accuracy and scene understanding in dynamic environments. Meanwhile, precise tracking of surrounding objects is essential for applications such as autonomous driving, augmented reality (AR), and virtual reality (VR). To this end, we propose LVIMOT, a tightly coupled LVI localization and multi-object tracking (MOT) system based on factor graph optimization, capable of jointly estimating the trajectories of the ego-vehicle and surrounding objects. In the proposed method, tracked objects are represented by 2D/3D bounding boxes and continuously associated by combining LiDAR and visual detections with multimodal feature cues. Building upon this, a binary hypothesis testing method for motion status classification is employed to identify object motion status by fusing motion and appearance information. Subsequently, the measurements associated with dynamic objects are fully exploited to construct object-related factors, which are jointly optimized with static features and IMU pre-integration factors within a unified factor graph to refine the trajectories of both the ego-vehicle and tracked objects. Extensive experiments on highly dynamic sequences from the KITTI and nuScenes datasets demonstrate that LVIMOT achieves state-of-the-art performance, with an HOTA of 80.37 for MOT, and the lowest mean translational and rotational absolute trajectory errors (ATE) for self-localization on KITTI (0.36 m, 0.02 rad) and nuScenes (0.13 m, 0.02 rad). The results confirm the effectiveness of integrating object-aware motion constraints and multimodal fusion in improving localization robustness and precision. The source code will be available at https://github.com/shqfeng/LVIMOT.git.