White-box structure analysis of pre-trained language models of code for effective attacking

Abstract

Context:

Pre-trained language models of code (PLMs-C for short) have dramatically improved the state-of-the-art on various programming language processing tasks.

Objective:

Due to these well-performed models being easily disturbed by slight perturbations, several black-box approaches have been proposed for attacking them. However, these studies have presented two challenges: (1) black-box attacks lack interpretability in generating adversarial examples and are inefficient in attacking; (2) white-box analysis methods in natural language processing (NLP) have not yet been applied to PLMs-C, incurring a gap in this field.

Methods:

To address these challenges, we make the first attempt to perform a white-box structure analysis for PLMs-C, followed by a grey-box attack for PLMs-C named MindAC based on derived linguistic structures. Specifically, referring to the probing tasks for analyzing the linguistic property of text in NLP, we first design eight novel probing tasks for code to perform white-box structure analysis. We derive three types of linguistic structures from PLMs-C named SurStruct, SyntStruct, and SemStrcut which correspond to the surface, syntactic and semantic structures of code, respectively. Subsequently, MindAC perturbs the code snippets through variable replacement, variable redefinition, and equivalent transformation of loop statements. Besides, in linguistic structures, MindAC introduces the angular distance of hidden output (ADHO) and the Euclidean distance of attention output (EDAO) to guide the generation of adversarial examples.

Results:

Our experiments reveal that: (1) for the first time, it has been demonstrated that PLMs-C possess three linguistic structures; (2) MindAc outperforms the state-of-the-art baselines on attack success rate by 2.47% $\sim$ 25.45%, reduces the execution time by 25 m $\sim$ 36h16 m, and achieves a significantly lower number of queries. Furthermore, we perform adversarial fine-tuning on the training sets and recover the Accuracy and F1 of the victim models by at least 57.76% and 60.13%, respectively.

Conclusion:

The results show that based on the derived linguistic structures, the proposed MindAC is more interpretable, effective, and efficient in attacking the PLMs-C compared with the state-of-the-art baselines. Besides, the generated adversarial examples can help to enhance the robustness of PLMs-C.