Anomaly detection using smart tracing tricks on call stack

Abstract

The call stack is an important baseline to detecting the intrusions spread over the system application programs penetrate and injected with malicious programs, also exploited by unauthorized users. But the previous work presented based on stack with the long training period, so in this paper demonstrate the extraction of sequences of return addresses generated by function calls in the code. This approach use two sets of input test data like return address set and function call sequence (virtual path) set. We apply smart trace tool and it is easy for anomaly detection and finding the unknown coding exploits as anomaly. We tested 14 attacks on Linux platform by setting different threshold values while training and given the affect of this technique with discussions on false positive rate.