Efficient Partial Weight Update Techniques for Lightweight On-Device Learning on Tiny Flash-Embedded MCUs

Abstract

Typical training procedures involve read and write operations for weight updates during backpropagation. However, on-device training on microcontroller units (MCUs) presents two challenges. First, the on-chip SRAM has insufficient capacity to store the weight. Second, the large flash memory, which has a constraint on write access, becomes necessary to accommodate the network for on-device training on MCUs. To tackle these memory constraints, we propose a partial weight update technique based on gradient delta computation. The weights are stored in flash memory, and a part of the weight to be updated is selectively copied to the SRAM from the flash memory. We implemented this approach for training a fully connected network on an on-device MNIST digit classification task using only 20-kB SRAM and 1912-kB flash memory on an MCU. The proposed technique achieves reasonable accuracy with only 18.52% partial weight updates, which is comparable to state-of-the-art results. Furthermore, we achieved a reduction of up to 46.9% in the area-power-delay product compared to a commercially available high-performance MCU capable of embedding the entire model parameter, taking into account the area scale factor.