Game-theoretic design of an information exchange model for detecting packed malware

Packing, a method used by the ‘good guys’ to protect their software from reverse engineering, is also used by the ‘bad guys’ to hide malicious code from being detected by anti-virus (AV) scanners. The AV industry is developing a mechanism to blacklist the software vendors that pack malicious applications, instead of the current practice of blacklisting the packers that are used for packing malicious applications. This will require packer developers to introduce ‘taggants’ in the packed executable and share taggant information in an industry wide information exchange. The idea is similar to the effort of requiring special chemicals to aid in the detection and identification of explosives. In the software context, it is expected that a packer vendor will introduce some secure watermark or signature that can identify the author of a packed binary, and hence help with the detection of malware. For a packer vendor to take on the extra work, which may cost him some customers, the AV industry may need to provide some incentive. However, since a packer vendor is an independent company, likely residing in a different legal jurisdiction, the AV industry cannot verify whether the packer vendor is indeed abiding by the terms of the incentive, and not selling a non-taggant version to malware authors through another channel. We use a game-theoretic modeling approach called the “principal-agent problem” to model the interaction between the AV industry and a packer vendor and give a method of computing the optimal incentive for packer vendors to tag and abide by the terms of the incentive.