TEESlice: slicing DNN models for secure and efficient deployment

Providing machine learning services is becoming profit business for IT companies. It is estimated that the AI-related business will bring trillions of dollars to the global economy. When selling machine learning services, companies should consider two important aspects: the security of the DNN model and the inference latency. The DNN models are expensive to train and represent precious intellectual property. The inference latency is important because modern DNN models are usually deployed to time-sensitive tasks and the inference latency affects the user's experience. Existing solutions cannot achieve a good balance between these two factors. To solve this problem, we propose TEESlice that provides a strong security guarantee and low service latency at the same time. TEESlice utilizes two kinds of specialized hardware: Trusted Execution Environments (TEE) and existing AI accelerators. When the company wants to deploy a private DNN model on the user's device, TEESlice can be used to extract the private information into model slices. The slices are attached to a public privacy-excluded backbone to form a hybrid model that has similar performance to the original model. When deploying the hybrid model, the lightweight privacy-related slice is secured by the TEE and the public backbone is put on the AI accelerators. The TEE provides a strong security guarantee on the model privacy and the accelerators reduce the computation latency of the heavy model backbone. Experimental results show that TEESlice can achieve more than 10x throughput promotion with the same level of strong security guarantee as putting the whole model inside the TEE. If the model provider wants to further verify the correctness of the accelerator's computation, TEESlice can still achieve 3-4x performance improvement.