Design and Implementation of OOM Module based on Rust

Abstract

The Linux kernel plays an important role in various application scenarios such as computers, mobile devices, and vehicles, in which security-sensitive information is stored. Undefined behavior in the C programming language is usually the cause of Linux kernel vulnerabilities, among which memory safety vulnerabilities are the most threatening. At present, the methods based on static/dynamic analysis and runtime software and hardware defense to ensure memory safety have the problems of poor performance, false positives, and poor compatibility. This paper presents the design and implementation of the OOM (out of memory) module based on the safe programming language Rust. We leverage the Rust FFI mechanism to design a foreign interface layer and a safe foreign interface layer to enable the reconstructed OOM module to invoke other Linux functionalities, and then use Rust to reconstruct the OOM module. The safety features of Rust language are used to avoid undefined behaviors, so as to improve the safety of the kernel. Finally, we export the C interface of the module to enable the invocation by the Linux kernel. Tests show that the OOM module performance and memory consumption after reconstruction are comparable to the original module.