Featherlight Stateful WebAssembly for Serverless Inference Workflows

Abstract

In serverless inference, complex prediction tasks are executed as workflows, relying on efficient state transfer across multiple functions. Serverless platforms typically deploy each function in a separate stateless container, depending on external processes for state management, which often results in suboptimal system utilization and increased latency. We introduce WasmFlow, a novel framework designed for serverless inference that ensures low latency and high throughput. This is achieved through process-level virtualization using WebAssembly. WasmFlow operates functions on a per-thread basis within compact WebAssembly modules, significantly reducing startup times and memory usage. The framework has two key features. (1) Efficient Memory Sharing: WasmFlow facilitates direct and rapid state transfer between functions using threads within the WebAssembly runtime. This is enabled through lightweight, lock-free, zero-copy intra-process communication, complemented by effective inter-process RPC. (2) System Optimizations: We further optimize WasmFlow with an advanced synchronization technique between functions, an affinity-aware workflow scheduler, and adaptive request batching. Implemented and integrated within the Kubernetes ecosystem, WasmFlow’s performance was evaluated using synthetic workloads and real-world Azure traces, including typical serverless workflows and ML models. Our results demonstrate that WasmFlow dramatically outperforms existing serverless frameworks. It reduces P90 end-to-end latency by 74x and 78x, increases function density by 1.7x and 223x compared to Faasm and SPRIGHT, and improves system throughput by 12.3x and 8.8x over Knative and WasmEdge, respectively.