Smart-hopping: Highly efficient ISA-level fault injection on real hardware

Fault-injection experiments on the instruction-set architecture level are commonly used to analyze embedded software's susceptibility to hardware faults, typically involving a vast number of experiments with systematically varying fault locations and times. Determinism and high performance are the predominant requirements on fault-injection platforms. Injecting faults into a real embedded hardware platform instead of a simulator is favorable for both workload execution speed and result accuracy. The most performance-critical part of such a fault-injection platform is the “fast forward” operation, which executes the target machine code without faults until the exact dynamic instruction is reached at which the execution must be stopped to inject the next fault. Unfortunately, most embedded CPUs do not support this operation efficiently. In this paper we present an approach that speeds up fast-forwarding significantly for most workloads with minimal requirements on hardware support. Based on a previously recorded instruction trace - which is needed for systematic fault-injection experiment planning anyways - we use standard debugging hardware to advance to a chosen point in program execution with a minimal number of steps. We evaluate our FAIL* tool platform with two MiBench benchmark categories, and improve experiment throughput by up to several magnitudes compared to similar fault-injection tools in the field.