Implementation and Optimization of Neural Networks for Tiny Hardware Devices

Traditionally, neural network inferencing on tiny hardware devices took place in a centralized server-based manner. With more real-time applications coming into play, where security and latency are a concern, there has become a need to move inferencing to the edge. This paper describes a machine learning pipeline to carry neural networks from their initial forms to compressed forms deployable on tiny hardware devices, while maintaining acceptable accuracies of the optimized models. We will review the different software optimization techniques used to compress neural networks to their deployable forms. The prototype is a proof of concept showing that applying knowledge distillation from a highly accurate ResNet20 model to a simple CNN student model, followed by post-training quantization, achieves good multi-class accuracy on a constrained Arduino Nano 33 BLE Sense at low power consumption and with low inferencing latency.