CECO-LAD: Cloud-Edge Collaboration for Unsupervised Log Anomaly Detection

Abstract

Artificial intelligence (AI)-driven log anomaly detection (LAD) is a critical component for maintaining the security and reliability of cyber infrastructure. However, deploying an effective LAD system in real-world environments presents a significant challenge, the cloud-edge dilemma, where accurate deep learning models favor centralized cloud resources, but operational constraints (e.g., latency, bandwidth, privacy, and energy) favor edge-local analysis. To address these challenges, we propose a comprehensive collaborative framework for heterogeneous cloud-edge LAD (CECO-LAD) that balances detection accuracy with resource efficiency. In CECO-LAD, we use expectation-maximization-enhanced anomaly transformer (EM-AT) as the base learner for the novel ensemble learning model, Bagging-style AT (BAT), the cloud-side ensemble detector, and further develop quantized BAT (Q-BAT) for resource-constrained edge environments. A Mahalanobis distance-based routing policy enables cloud-edge collaboration by selectively forwarding only uncertain samples to the cloud and retaining confident cases at the edge, thereby minimizing resource consumption while maximizing detection accuracy. Additionally, we propose Green-LADE, a Green AI-inspired method to enable holistic edge-side evaluation. Experiments on the Hadoop Distributed File System (HDFS), OpenStack, and blue gene/l (BGL) datasets show that CECO-LAD achieves state-of-the-art anomaly detection performance while substantially reducing resource consumption on resource-constrained edge devices. CECO-LAD achieves $F_{1}$-scores of 99.98% on HDFS, 99.95% on OpenStack, and 99.99% on BGL. The proposed framework advances LAD by making it accurate, deployable, and sustainable.