Malware Detection Using Gradient Boosting Decision Trees with Customized Log Loss Function

The increasing number of malicious software spread through the Internet has become a serious threat. Malware authors use obfuscation and deformation techniques to generate new types of malware to evade the detection of traditional detection methods so that it is widely expected for machine learning methods that classify malware and cleanware based on the characteristics of the samples. The current research trend is to use machine learning technology, especially decision tree technology, to identify new malicious software quickly and accurately. The purpose of this paper is to investigate malware classification accuracy based on the latest decision tree-based algorithms with a custom log loss function. Therefore, we use the FFRI Dataset 2019 to construct baseline malware detection models from surface analysis logs and PE header dumps. Then, we customize a classification log loss function, makes an 82% reduction of false positives with sacrificing twice false negatives. To keep malware detection covering and quick countermeasure to true positive results, we propose a hybrid usage of normal log loss function model and custom log loss function model to give additional priority to positive results.