Flex Tuner: A Flexible Container-Based Tuning System for Cloud Applications

Abstract

With the rapid growth of dataset, more and more traditional applications turn into big data applications running on cloud platforms. This trend has continued, and even accelerated. Compared with traditional applications, big data applications spend lots of time to transfer data among computing nodes. Hence, the communication optimization is a must for big data applications. Network topology and routing algorithm of the underlying system are two major factors in determining the communication performance of big data applications. Once the system is deployed, the network topology is fixed and static or dynamic routing protocols are preinstalled. Users cannot change them. Therefore, it is hard for application developers to identify the optimal network configuration for their applications with distinct communication patterns. In this study, we design a flexible container-based tuning system (Flex Tuner) allowing users to create a farm of lightweight virtual machines (containers) on host machines. In addition, we use software-defined networking (SDN) technique to connect and direct the network traffic among these containers. Users can soft-tune the network topology and network traffic of the Flex Tuner, thereby enabling application developers to analyze their applications on the same system with different network configuration. The preliminary experimental results have shown that Flex Tuner can represent application performance variations caused by network topology and routing algorithm. Case studies through both synthetic big data programs and benchmarks have indicated that Flex Tuner enables researchers to analyze the communication cost of their big data applications and to find the suitable network topology and routing algorithm.