MMBypass: Towards efficient multi-modal AI computing with adaptive bypass network

Abstract

Multi-modal artificial intelligence systems demonstrate superior performance through cross-modal information fusion and processing mechanisms, surpassing conventional unimodal architectures. However, the enhanced computational complexity required for processing heterogeneous data streams in multi-modal frameworks results in elevated inference latency compared to their uni-modal architectures. This limitation significantly constrains deployment feasibility for real-time and large-scale applications. To address this challenge, we present MMBypass, an adaptive and efficient architecture for multi-modal AI acceleration. Our solution implements intelligent layer-skipping mechanisms through adaptive computational complexity analysis of multi-modal tasks, achieving latency reduction while maintaining predictive accuracy and mitigating model overfitting in specialized scenarios. The architecture's innovation lies in two aspects: 1) We design bypasses for each uni-modal network in multi-modal networks to perform adaptive computing. 2) We design a guider to dynamically choose the optimal bypasses. Distinct from existing methods, MMBypass maintains broad applicability without requiring domain-specific prerequisites, and it shows significantly better performance on data samples with different difficulties. Empirical evaluations demonstrate our architecture achieves 44.5% average latency reduction while matching or exceeding baseline accuracy across diverse multi-modal benchmarks.