Understanding practitioners’ challenges and requirements in the design, implementation, and evaluation of anti-phishing interventions

Abstract

Background:

Research shows that the ineffectiveness of anti-phishing interventions can result from practitioners’ failure to consider end-users’ requirements in the intervention design, implementation, and evaluation. To assist practitioners in addressing usability issues, we reported 41 guidelines through a systematic Multi-vocal Literature Review (MLR). The usefulness of these guidelines in real-world scenarios remains uncertain until the involved challenges and requirements to implement them are investigated.

Objective:

(1) To investigate practitioners’ challenges in the design, implementation, and evaluation of phishing interventions in real-world settings; (2) to understand practitioners’ perspectives on our guidelines and how they can be made easily accessible to the practitioners.

Method:

We interviewed 18 practitioners (intervention designers, security practitioners, and C-suite employees) from 18 organizations in 6 countries.

Results:

(1) We identify 8 challenges in training content design, anti-phishing datasets, post-training knowledge assessment, and so on. We compare these challenges with the challenges identified from our MLR to demonstrate the ecological validity of the challenges found in MLR and derive a set of insights to overcome them; (2) we report practitioners’ feedback on our guidelines; (3) we gather actionable features on an envisioned tool to make these guidelines easily accessible. Conclusion: We provide 15 recommendations to improve the anti-phishing defense in the organisations.