APIARY: An API-based automatic rule generator for yara to enhance malware detection

Abstract

Cyber threats, primarily malware, have increased with rapid technological advancements in various fields. This growing complexity requires sophisticated and automated malware detection tools because traditional methods cannot keep up with the sheer volume of threats and their evolution. Detection mechanisms that are resilient against evolved malware behaviors, which are typically described by application programming interface (API) functions, are essential for real-time system protection. This paper presents APIARY, an innovative API-based Automatic Rule generator for the YARA tool, designed to enhance malware identification through customized signatures based on peculiar API-based patterns. It discovers distinctive APIs that distinguish malware from goodware, regardless of input data coming from dynamic and static analyses of Windows-like executable files. The algorithm assigns relevance scores to each variable and discards less significant features to identify critical malware indicators. In addition, the generation process optimizes the identified malware model categories to increase the detection rate while minimizing the number of rules produced. The experimental results obtained on nine datasets sourced from the literature demonstrate the potential of APIARY to automatically produce highly effective YARA rules in a short time. Moreover, the rules generated outperform those obtained using alternative state-of-the-art algorithms in terms of detection performance. Lastly, unlike competitors, the proposed procedure does not rely on additional malware analysis data, such as network connection attempts or API parameters, achieving a more streamlined and efficient detection process.