DyCE: Dynamically Configurable Exiting for deep learning compression and real-time scaling

Conventional deep learning (DL) model compression methods affect all input samples equally. However, as samples vary in difficulty, a dynamic model that adapts computation based on sample complexity offers a novel perspective for compression and scaling. Despite this potential, existing dynamic techniques are typically monolithic and have model-specific implementations, limiting their generalizability as broad compression and scaling methods. Additionally, most deployed DL systems are fixed, and unable to adjust once deployed. This paper introduces DyCE, a dynamically configurable system that can adjust the performance-complexity trade-off of a DL model at runtime without needing re-initialization or re-deployment. DyCE achieves this by adding exit networks to intermediate layers, thus allowing early termination if results are acceptable. DyCE also decouples the design of exit networks from the base model itself, enabling its easy adaptation to new base models. We also propose methods for generating optimized configurations and determining exit network types and positions for dynamic trade-offs. By enabling simple configuration switching, DyCE enables fine-grained performance-complexity tuning in real-time. We demonstrate the effectiveness of DyCE through image classification tasks using deep convolutional neural networks (CNNs). DyCE significantly reduces computational complexity by 26.2% for ResNet${}_{\mathrm{152}}$, 26.6% for ConvNextv2${}_{\mathrm{tiny}}$ and 32.0% for DaViT${}_{\mathrm{base}}$ on ImageNet validation set, with accuracy reductions of less than 0.5%.