TinyMo: Graph-Level Memory Optimizer for Tiny Machine Learning

Abstract

Effective memory optimization is essential for tiny machine learning because tiny embedded systems generally have limited memory for model execution. Previous research has proposed various model compression methods to reduce the memory usage of machine learning models. However, the methods often entail accuracy loss resulting from altering model weights. This letter proposes a graph-level memory optimizer for tiny embedded systems, TinyMo, which optimizes the memory usage of an input model by changing the structure of the model graph. TinyMo mainly uses two optimization methods: 1) tensor spilling and 2) tensor splitting, to reduce unnecessary memory usage from long skip connections and large separable convolutions. In the evaluation, this letter shows that the proposed optimizer can successfully reduce the peak memory usage of various neural network models for commercial embedded systems with little runtime overhead.