Discovering instructions for robust binary-level coverage criteria

Abstract

Object-Branch Coverage (OBC) is often used to measure effectiveness of test suites, when source code is unavailable. The traditional OBC definition can be made more resilient to variations in compilers and the structure of generated code by creating more robust definitions. However finding which instructions should be included in each new definition is laborious, error-prone, and architecture-dependent. We automate the discovery of instructions to be included for an improved OBC definition on the X86 and ARM architectures. We discover all possible valid instructions by symbolically executing instruction decoders for X86 and ARM instructions. For each discovered instruction, we translate it to Vine IR, and check if the Vine IR translation satisfies the OBC definition. We verify the correctness of our tool by comparing its output with the X86 and ARM architecture manuals. Our automated instruction classification facilitates development of more robust OBC definitions with better bug-finding ability and lesser sensitivity to compiler variations.