Towards an Operations-Aware Experimentation Methodology

Abstract

Security Operations Centers (SOCs) serve a critical role in protecting enterprise networks and systems. Despite this critical role, only a limited number of researchers in the field have an awareness of the obstacles and challenges in applying cyber ranges and cybersecurity testbeds to the area of SOC training, exercises and evaluation. This paper introduces a systematic approach to incorporating SOCs into cybersecu-rity experiments, including both training and evaluation. We present a reference SOC model, an implementation of that model and downloadable software distributions suitable for deploying on cyber ranges, and guidance towards a methodol-ogy to promote rigorous experiments including those involving human cyber operators. Metrics focused on analyst event load are presented in the context of measuring the impact of new threats, technologies and procedures on SOC performance. Collectively, these contributions serve as a basis for future work to engage the research and operational communities to work together to advance the state-of-the-art of SOC technologies and SOC operators.