Establishing trust in black-box programs

Abstract

Encrypted binaries are increasingly being used as deterrence for software piracy as well as vulnerability exploitation. The application of encrypted programs, however, leads to increased security concerns, as users are unable to identify malicious behavior by monitoring the encrypted executables. This paper proposes a method to monitor encrypted programs that assures users that the black-box program on their system is not violating any security concerns. Our approach is to embed a system call monitoring tool into the operating system that monitors system call content for suspicious behavior or the lack thereof.