TI-Prompt: Towards a Prompt Tuning Method for Few-shot Threat Intelligence Twitter Classification*

Abstract

Obtaining the latest Threat Intelligence (TI) via Twitter has become one of the most important methods for defenders to catch up with emerging cyber threats. Existing TI Twitter classification works mainly based on supervised learning methods. Such approaches require large amounts of annotated data and are difficult to be transferred to other TI Twitter classification tasks. This paper proposes a prompt-based method for classifying TI on Twitter, named TI-Prompt. TI-Prompt lever-ages the prompt-tuning method with two templates in different TI Twitter classification tasks. TI-Prompt also uses a semantic similarity-based approach to automatically enrich the prompt verbalizer without expert knowledge and a verbalizer refinement method to calibrate the verbalizer based on the training data. We evaluate TI-Prompt with binary and multi-classification tasks on two Twitter Threat Intelligence datasets. Evaluation results show that the proposed TI-Prompt improves 5-10% over the best performance of previous supervised learning methods under the few-shot settings. Compared to the general prompt-tuning methods, the proposed prompt-tuning templates can also improve the classification performance by 2–5%. Meanwhile, the proposed verbalizer enrichment method and refinement method improve classification accuracy by 1–4% compared with the general single-word verbalizer prompt method. Therefore, TI-Prompt can be extended to other Threat Intelligence classification tasks without requiring large amounts of training data, significantly reducing the annotation cost.