Elastic Cross-Layer Orchestration of Network Policies in the Kubernetes Stack

Abstract

Packaging applications in Containers, dynamically managed using a cluster orchestrator, is the de-facto approach for deployment of cloud-native applications. When Containers run inside Virtual Machines (VMs) to protect infrastructural assets, Network Policies at the Container layer and Security Groups at the VM layer provide complementary firewall mechanisms that strengthen defenses against lateral movement of attackers. However, least-privilege network policies at the Container layer may not always be consistent with statically defined, over-permissive Security Groups at the VM layer. This is especially a problem with low-latency configuration of Container networking solutions that requires every opened Container protocol, port and traffic direction also to be opened at the VM layer. In any post-exploitation scenario where attackers escape from within an already compromised or infected Container, such over-permissive Security Groups do not prevent the attacker from spreading across VMs to find powerful tokens for accessing the cluster orchestrator. In this paper, we introduce GrassHopper, a fast and dynamic cross-layer enforcement approach for Network Policies, which automatically generates Security Group configurations from dynamically verified Network Policies and Container scheduling decisions. Given the low-latency context, the design of GrassHopper must ensure that dynamically generated Security Group rules come in a timely manner to effect before the newly scheduled Containers become ready to serve traffic. We evaluate the performance of GrassHopper on a Kubernetes cluster running on OpenStack at the network and application level. In comparison to a Security Group management approach that is not scheduling-aware, our findings show that for low-latency applications GrassHopper can reduce the network attack surface between VMs at a ratio of 78-to-99%, while causing no network performance overhead at the application level with respect to latency and throughput.