A Unified Accelerator for All-in-One Image Restoration Based on Prompt Degradation Learning

Abstract

All-in-one image restoration (IR) recovers images from various unknown distortions by a single model, such as rain, haze, and blur. Transformer-based IR methods have significantly improved the visual effects of the restored images. However, deploying complex IR models on edge devices is challenging due to massive parameters and intensive computations. Moreover, existing accelerators are typically customized for a single task, resulting in severe resource underutilization when executing multiple tasks. Therefore, this paper develops an algorithm-hardware co-design framework to accelerate a novel CNN-Transformer cooperative model for multiple IR tasks. Firstly, on the algorithm level, an Efficient Restoration Foundational Model (ERFM) is proposed to recover corrupted images from various degradations with low model complexity. Secondly, to guide adaptive corruption removal, a novel prompt learning scheme is introduced to fuse context-related degradation cues and boost high-quality reconstruction. Thirdly, on the hardware level, an integer approximation method is proposed to avoid expensive hardware overhead caused by complex nonlinear operations, such as layer normalization and softmax while maintaining comparable IR quality. Moreover, a head stationary dataflow and softmax fusion mechanism are designed to reduce data movement and enhance on-chip resource utilization. Finally, an overall hardware architecture is developed and implemented in TSMC 28 nm CMOS technology. Experimental results show that our ERFM achieves better visual perception than other baselines on seven challenging IR tasks without task-specific fine-tuning. Moreover, compared to other accelerators for vision Transformers, our design can achieve $3.3\times $ and $3.7\times $ improvements in throughput and energy efficiency.