An open testbed for O-RAN experimentation with AI-enabled control and monitoring

The proliferation of open radio access networks (O-RAN) in modern 5G systems has ushered in enhanced flexibility and efficiency, but it has also introduced novel security challenges. In response, this paper presents a novel AI-based anomaly detection framework tailored for O-RAN networks operating in 5G environments. By employing principal component analysis for dimensionality reduction and a deep neural network for classification, the proposed system efficiently processes large-scale 5G traffic data while achieving high detection accuracy and low latency. Experimental evaluation on an open-source testbed with realistic cellular traffic demonstrates rapid convergence, with both training and validation accuracy values approaching 100% and effective detection of anomalies introduced via user equipment identifier swaps. The testbed processed over 300,000 traffic samples with 31 distinct network features, emulating 8 unique user equipment profiles under diverse radio conditions. Under adversarial scenarios, such as identity-swapping attacks, the system identified anomalous behavior with detection rates exceeding 40%, while maintaining a near-zero false positive rate on clean traffic. These results underscore the testbed’s capability to simulate complex 5G environments and the framework’s ability to deliver highly accurate, low-latency, and scalable anomaly detection. Overall, this work highlights the potential of advanced AI techniques to significantly enhance the security and resilience of modern wireless communication networks.