NDIF: A distributed framework for efficient in-network neural network inference

In-network machine learning is a promising technology that offloads machine learning models onto programmable data planes to enable intelligent decision-making by programmable devices. Such advancement empowers security applications (e.g., intrusion detection) to adapt to dynamic network changes in real time and make rational decisions. Existing research deploys neural network models in a distributed way on programmable data planes, with the aim of performing real-time inference using network-wide compute resources. However, existing research primarily focuses on model implementations, with little attention paid to the negative impact on the efficiency and robustness of in-network applications introduced by the inference process. We propose NDIF, a framework for performing in-network neural network inference in a distributed manner. NDIF enables in-network inference on arbitrary programmable devices, with each device autonomously managing its inference workload based on available resources. Moreover, new inference schemes can be easily deployed by writing entries into programmable devices to adapt to network changes. These benefits improve the efficiency and stability of the in-network inference process, thereby enhancing the efficiency and robustness of in-network applications built based on neural network models. The experiments on the use cases of anomaly detection and packet classification demonstrate that NDIF outperforms previous inference frameworks across various quality of service (QoS) metrics while maintaining a reasonable cost.