A Framework for Consistent and Repeatable Controller Area Network IDS Evaluation

—The landscape of automotive vehicle attack surfaces continues to grow, and vulnerabilities in the controller area network (CAN) expose vehicles to cyber-physical risks and attacks that can endanger the safety of passengers and pedestrians. Intrusion detection systems (IDS) for CAN have emerged as a key mitigation approach for these risks, but uniform methods to compare proposed IDS techniques are lacking. In this paper, we present a framework for comparative performance analysis of state-of-the-art IDSs for CAN bus to provide a consistent methodology to evaluate and assess proposed approaches. This framework relies on previously published datasets comprising message logs recorded from a real vehicle CAN bus coupled with traditional classifier performance metrics to reduce the discrepancies that arise when comparing IDS approaches from disparate sources.