Dependability and Protection of Transformer Models Against Soft Errors on Text Embeddings

Abstract

Transformers have achieved remarkable success in diverse fields such as Natural Language Processing (NLP) and computer vision (CV). For pre-trained Transformer models involving text processing, embedding representations are important parameters, incurring a large volume of memory. Soft errors on embedding vectors can lead to incorrect inputs to Transformers, and if not corrected in time, accumulated errors may produce undesirable outcomes. This paper considers the dependability of text related Transformer models to accumulated errors on embedding parameters and takes three typical models in different applications as case studies: BERT based sentence emotion classification, T5 based text summarization, and CLIP based image classification. We first evaluate the dependability of the three models by injecting bit errors on embedding parameters; only errors on a few critical bits affect model performance. Based on this finding, we first propose an efficient selective protection for embedding parameters with small values, and then through scaling, we extend the scheme for models with large embedding parameters. Extensive simulation results show that the proposed protection scheme can effectively remove the impact of soft errors on task performance. In particular, the complexity overhead of the proposed scheme is negligible, and the additional memory overhead as encountered in the SEC scheme is avoided.