Optimising AI models for intelligence extraction in the life cycle of Cybersecurity Threat Landscape generation

Abstract

The increasing complexity and frequency of cyber attacks in the modern digital environment demand continuous vigilance and proactive strategies to manage risks effectively. Conventional approaches to generating intelligence for Cybersecurity Threat Landscape (CTL) reports are often resource-intensive and time-consuming, as they depend on manual identification, collection, and analysis of relevant electronically stored information (ESI). This study investigates the potential of artificial intelligence (AI) to transform CTL generation, reducing manual classification and tagging while improving efficiency and accuracy.

We focus on evaluating the classification performance of several Large Language Models (LLMs), including Gemini 1.5 Pro, GPT-4o, but also Bidirectional Encoder Representations from Transformers (BERT) based models like TRAM and TTPHunter along with custom Named Entity Recognition (NER) models, using a dataset previously annotated by human experts. Our findings demonstrate the promising results of AI-driven intelligence extraction for CTL report generation, streamlining cybersecurity operations by automating routine tasks and providing precise and timely threat intelligence. However, the variability in model performance suggests the importance of hybrid approaches needed to achieve the accuracy of human annotation. Therefore, we propose a novel voting agreement-based methodology, harvesting the most from the combined AI model capabilities to effectively address the complexities of cybersecurity threat intelligence extraction.