Masked variational transformer for complex clutter modeling and target detection

Weak target detection commonly encounters intense clutter interference, which overshadows weak signals and complicates the task. Taking advantage of the powerful data mining capability of neural networks, more and more deep learning-based methods are applied to radar target detection. Among the approaches, those founded upon unsupervised learning methodologies exhibit remarkable merit because they dispense with the requirement for target samples within the training step, making them highly applicable in practical target detecting scenarios. However, existing methods suffer from limitations in leveraging the range-Doppler (R-D) two-dimensional correlation and finely modeling in multiple clutter scenarios. In this paper, an unsupervised Transformer-based detector (TrDet) is proposed to break through the boundary of modeling capability. First, with the designed two-dimensional position embedding (2-DPE) and global query embedding (GQE) techniques, an unsupervised training strategy for R-D spectrum based on Transformer framework is utilized to achieve refined clutter modeling. Then, radar target detection is formulated as an out-of-distribution (OOD) detection task to mitigate clutter interference. Moreover, the masked variational Transformer-based detector (MVTrDet) is further proposed to prevent target information leakage when the target is in close proximity to the clutter in Doppler domain. Compared with several relative algorithms, our proposed methods are better suited for radar target detection in complex clutter environments. The experimental results derived from both measured data and simulated data verify the effectiveness of our proposed methods.