Measuring Learning in a Cyber Security Exercise

In recent years, cyber security exercises have established themselves as an integral part of cyber security education. Cyber security professionals usually work as a part of a team that monitors and responds to incidents in the environment. A sufficiently realistic complex learning environment is necessary for collaborative learning at the expert level. Evaluating the learning outcomes of complex exercises is an important task for both assessing how individuals met the learning objectives, and how to improve the exercise to better serve those goals. This requires the assessment of multiple skill and knowledge categories independently. We leveraged the NIST NICE Cybersecurity Workforce Framework as a base for building knowledge categories for questionnaire use. However, the NICE framework is comprehensive and detailed requiring that the areas of competence assessment needed to be simplified for questionnaire use. We summarized the NICE framework into 44 questions addressed to the individuals who participated in the exercise. A web-based questionnaire was used to query 21 participants’ skill level before and after the exercise, as well as their familiarity and experience with the topic before and during the exercise. The results indicate that cyber security exercises will increase the knowledge of the participant in the knowledge areas that were present in the exercise. This increase was more prominent in cases where the participant was more likely to recognize, and experience events related to that category during the exercise. Furthermore, we concluded that the NICE framework can be used to assess individual know-how and as a basis for knowledge-related questionnaires.