A Robust Image Semantic Communication System With Multi-Scale Vision Transformer

Abstract

Semantic communications have demonstrated exceptional performance across various tasks, yet they are susceptible to semantic impairments due to the inherent vulnerability of deep neural networks. This paper focuses on semantic impairments in images, particularly those stemming from adversarial perturbations. We introduce a novel metric for quantifying the level of semantic impairment and create a semantic impairment dataset. Furthermore, we propose a deep learning enabled semantic communication system for robust image transmission, termed as DeepSC-RI. The proposed system harnesses a multi-scale semantic extractor with a dual-branch design tailored for extracting semantics with varying granularity, thereby boosting the robustness of the system. The fine-grained branch incorporates a semantic importance evaluation module to identify and prioritize crucial semantics through self-attention score manipulations, while the coarse-grained branch adopts a hierarchical approach for progressively capturing the robust semantics. These two streams of semantics are seamlessly integrated via an advanced cross-attention-based semantic fusion module. Experimental results highlight the superior performance of DeepSC-RI under diverse channel conditions, across various levels of semantic impairment intensity, and in multiple tasks.