Understanding How National CSIRTs Evaluate Cyber Incident Response Tools and Data: Findings from Focus Group Discussions

National Computer Security Incident Response Teams (CSIRTs) have been established worldwide to coordinate responses to computer security incidents at the national level. While it is known that national CSIRTs routinely use different types of tools and data from various sources in their cyber incident investigations, limited studies are available about how national CSIRTs evaluate and choose which tools and data to use for incident response. Such an evaluation is important to ensure that these tools and data are of good quality and, consequently, help to increase the effectiveness of the incident response process and the quality of incident response investigations. Seven online focus group discussions with 20 participants (all staff members) from 15 national CSIRTs across Africa, Asia Pacific, Europe, and North and South America were carried out to address this gap. Results from the focus groups led to four significant findings: (1) there is a confirmed need for a systematic evaluation of tools and data used in national CSIRTs, (2) there is a lack of a generally accepted standard procedure for evaluating tools and data in national CSIRTs, (3) there is a general agreement among all focus group participants regarding the challenges that impinge a systematic evaluation of tools and data by national CSIRTs, and (4) we identified a list of candidate criteria that can help inform the design of a standard procedure for evaluating tools and data by national CSIRTs. Based on our findings, we call on the cyber security community and national CSIRTs to develop standard procedures and criteria for evaluating tools and data that CSIRTs, in general, can use.