A DNN Inference Offloading Scheme for Storage Arrays

Abstract

Recent advancement in deep learning technology has brought tremendous amounts of deep neural network (DNN) inference jobs into a data center. While hardware accelerators for DNN computations have made rapid progress, network capability to transfer a large amount of data needed for DNN computations still is a common bottleneck threatening service level objectives (SLO). To alleviate such a bottleneck occurred by data transfer, we propose a novel system architecture that offloads DNN inference job to a storage node. Our system includes concise API which mitigates the programming burden needed to offload computations, and software architecture to conduct general DNN inference jobs in a conventional storage system. Experimental results show that our system exhibits a 35% of shorter average latency and more than 99% reduction in network usage in common image retrieval and classification jobs over existing systems.