Hiddenite: 4K-PE Hidden Network Inference 4D-Tensor Engine Exploiting On-Chip Model Construction Achieving 34.8-to-16.0TOPS/W for CIFAR-100 and ImageNet

Since the advent of the Lottery Ticket Hypothesis [1], which advocates the existence of embedded sparse models that achieve accuracies equivalent to the original dense model, new algorithms to find such subnetworks have been attracting attention. In particular, Hidden Network (HNN) [2] proposed a training method that finds such accurate subnetworks (Fig. 15.4.1). HNN extracts the sparse subnetwork by taking a logical AND of an initial model's random weights and a binary mask that defines the selected connections - a supermask. The importance of each connection, quantified as a score, is evaluated in the training phase; a supermask is learned by picking the connections with the top-k% highest scores. Although similar to pruning, supermask training is clearly different in that it never updates the initial random weights.