Anomaly Detection in Clouds: Challenges and Practice

Cloud computing is an important infrastructure for many enterprises. After 10 years of development, cloud computing has achieved a great success, and has greatly changed the economy, society, science and industries. In particular, with the rapid development of mobile Internet and big data technology, almost all of the online services and data services are built on the top of cloud computing, such as the online banking services provided by banks, the electronic services provided by the news media, the government cloud information systems provided by the government departments, the mobile services provided by the communications companies. Besides, tens of thousands of Start-ups rely on the provision of cloud computing services. Therefore, ensuring cloud reliability is very important and essential. However, the reality is that the current cloud systems are not reliable enough. On February 28th 2017, Amazon Web Services, the popular storage and hosting platform used by a huge range of companies, experienced S3 service interruption for 4 hours in the Northern Virginia (US-EAST-1) Region, and then quickly spread other online service providers who rely on the S3 service [2]. This failure caused a huge economic loss. It is because cloud computing service providers typically set a Service Level Agreement (SLA) with customers. For example, when customers require 99.99% availability, it means that 99.99% of the time must meet the requirement for 365 days per year. If the service breaks more than 0.01%, compensation is required. In fact, with the continuous development and maturity of cloud computing, a large number of traditional business systems have been deployed on the cloud platform. Cloud computing integrates existing hardware resources through virtualization technology to create a shared resource pool that enables applications to obtain computing, storage, and network resources on demand, effectively enhancing the scalability and resource utilization of traditional IT infrastructures and significantly reducing the operation cost of the traditional business systems. However, with the growing number of applications running on the cloud, the scale of cloud data center has been expanding, the current cloud computing system has become very complex, mainly reflected in: 1) Large scale. A typical data center involves more than 100,000 servers and 10,000 switches, more nodes usually mean higher probability of failure; 2) Complex application structure. Web search, e-commerce and other typical cloud program has a complex interactive behavior. For example, an Amazon page request involves interaction with hundreds of components [7], error in any one component will lead to the whole application anomalies; 3) Shared resource pattern. One of the basic features of cloud computing is resource sharing, a typical server in Google Cloud data center hosts 5 to 18 applications simultaneously, each server runs about 10.69 applications [5]. Resource competition will interfere with each other and affect application performance. The complexity of these cloud computing systems, the complexity of application interaction structure and the inherent sharing pattern of cloud platforms make cloud systems more prone to performance anomalies than traditional platforms. It can be said that anomaly is a normal state in cloud computing [3]. For further analysis, resource competition, resource bottlenecks, misconfiguration, software defects, hardware failures, and external attacks can cause cloud system anomalies or failures. Performance anomaly refers to any sudden degradation of performance that deviates from the normal behavior of the system. Unlike outages that cause the system to stop running immediately, performance anomalies typically result in a decrease in system efficiency. The reasons such as misconfiguration, software defects, hardware failures, can often cause performance anomalies. For cloud computing systems, it is not enough to detect outages or other functional anomalies, because those anomalies often cause service interruption and can be resolved by simply restarting or replacing hardware. While performance anomalies caused by resource sharing and interference are more worthy of attention [4], because the performance anomalies can be eliminated before service interruption to ensure continued services. If the performance anomalies of cloud computing system are not timely handled, it may cause very serious consequences, which not only affect the business system to run normally, but also hinder the enterprise to deploy their services on cloud systems. Especially for the those latency-sensitive cloud applications, it is extremely important to eliminate performance anomalies in a timely manner. For example, Amazon found a 1% decline in sales per 100ms latency, Google found a 20% drop in traffic for every 0.5s latency in search page, and stock traders found that it would cause a loss of 400 Million dollars if their electronic trading platform lagged behind the competitors by 5 ms. Other research also shows that the average maximum time of cloud data center failure is about 100 hours, which seriously affects the experience of cloud service users. In the cloud environment, as a large number of business systems are deployed in the cloud data center, cloud data center failure will affect a large number of users, such as the previously mentioned Amazon S3 failure, resulting in serious economic losses. Thus, timely and accurate detection of the cloud computing anomalies is very important. Anomaly detection is an effective means to help cloud platform administrators monitor and analyze cloud behaviors and improve cloud reliability. It helps to identify unusual behavior of the system so that cloud platform administrators can take proactive operations before a system crash or service failure. However, due to the characteristics such as large-scale, complex and resource sharing, it is very difficult to accurately detect anomalies in cloud computing. If the anomalies can not be accurately detected, the further recovery will be out of the question. Due to the importance of the problem, current mainstream cloud computing service providers usually provide online monitoring services. Amazon developed CloudWatch [1] for its EC2 users to monitor virtual machine instance status, resource usage, network traffic, disk read and write status, etc. Google developed Dapper framework [6] to provide state monitoring for Google search engines. But those monitoring services only provide simple data presentation, lack of in-depth analysis of monitoring data (such as cross-level correlation analysis), and is not intelligent enough for anomaly reasoning (such as cross-node fault source localization). In cloud data center, as the size of detected objects is very large and interrelated, the object being detected itself is in a high dynamic environment, it is very challenging to detect anomalies in an accurate, real-time and adaptive way. The existing anomaly detection solution lacks effective discovery and reasoning of the anomaly, which leads to the inability to locate and eliminate the anomaly in time. This is also the main reason that causes the current cloud platform accident frequently. In this talk, we introduce our solution for anomaly detection in clouds. 1) Anomaly detection. In order to efficiently detect the potential anomalies, we perform large-scale offline performance testing and also create an online detection method. i) Offline testing. The purpose is to find the key performance bottleneck and quantify comparison between difference hardware and software. We first propose a three-layer benchmarking methodology to fully evaluates cloud performance and then present a new benchmark suite -- Virt-B [11] -- that measures various scenarios, such as single machine virtualization, server consolidation, VM mapping, VM live migration, HPC virtual clusters and Hadoop virtual cluster. Finally, we introduce a performance testing toolkit to automate the benchmarking process. ii) Online detection. The purpose is to monitor applications in real time and quickly detect potential faults. We propose a quantile regression based online anomaly detection method and did a case study on 67 real Yahoo! anomaly traffic datasets. 2) Anomaly inference. We propose a dependency graph based anomaly inference method. Dependency reflects interaction relationship and execution path, and can be used for fault localization. There are usually three methods that can be used to fetch the dependency graph: instrumentation, extract configuration files and analyze network traffic. We create light-weight agents to monitor the traffic and use sampling technique to reduce the overheads. The advantages of our solution include: supports VMs (Xen/KVM), accuracy guarantee based on probability theory, dynamic dependency construction, focuses on the PM/VM layer, and low overheads. 3) Anomaly recovery. The traditional recovery methods like Checkpoint/Restart has high overheads and are not suitable for latency-sensitive applications. While we propose two solutions: cache-aware fault VM isolation and migration-based fault recovery. i) Cash-Aware Fault Isolation [8]. We first give a quantitative definition of isolation, then we propose a VM-core scheduling method to improve the fault isolation. ii) Fault Recovery based on Migration [9, 10, 12]. We propose a fault recovery method based on live migration. The main advantages include: online service, low overheads, and can be used in large-scale cloud datacenters.