SMART: Sim2Real Meta-Learning-Based Training for mmWave Beam Selection in V2X Networks

Abstract

Digital twins (DT) offer a low-overhead evaluation platform and the ability to generate rich datasets for training machine learning (ML) models before actual deployment. Specifically, for the scenario of ML-aided millimeter wave (mmWave) links between moving vehicles to roadside units, we show how DT can create an accurate replica of the real world for model training and testing. The contributions of this paper are twofold: First, we propose a framework to create a multimodal Digital Twin (DT), where synthetic images and LiDAR data for the deployment location are generated along with RF propagation measurements obtained via ray-tracing. Second, to ensure effective domain adaptation, we leverage meta-learning, specifically Model-Agnostic Meta-Learning (MAML), with transfer learning (TL) serving as a baseline validation approach. The proposed framework is validated using a comprehensive dataset containing both real and synthetic LiDAR and image data for mmWave V2X beam selection. It also enables the investigation of how each sensor modality impacts domain adaptation, taking into account the unique requirements of mmWave beam selection. Experimental results show that models trained on synthetic data using transfer learning and meta-learning, followed by minimal fine-tuning with real-world data, achieve up to 4.09× and 14.04× improvements in accuracy, respectively. These findings highlight the potential of synthetic data and meta-learning to bridge the domain gap and adapt rapidly to real-world beamforming challenges.