Security and 5G: Attack mitigation using Reinforcement Learning in SDN networks

5G ecosystem is shaping the future of communication networks enabling innovation and digital transformation not only for individual users but also for companies, industries, and communities. In this scenario, technologies such as Software Defined Networking (SDN) represent a solution for telecommunications providers to create agile, scalable, efficient platforms capable of meeting the requirements in the 5G ecosystem. However, as network environments and systems become increasingly complex, both in terms of size and dynamic behavior, the number of vulnerabilities in them can be very high. In addition, hackers are continuously improving intrusion methods, which are becoming more difficult to detect. For this reason, in this study, we deploy a system based on a Reinforcement Learning (RL) agent capable of applying different countermeasures to defend a network against intrusion and DDoS attacks using SDN. The approach is drawn like a serious game in which a defender and an attacker carry out actions based on the observations they get from the environment, i.e., network current status. In this study, defenders and attackers are trained using the Deep Q-Learning (DQN) algorithm with some variations, like Prioritized Replay, Dueling, and Double DQN, comparing their results in order to get the best strategy for attack mitigation. The results of this paper show that RL algorithms can be successfully used to create more versatile agents able of interpreting and adapting themselves to different situations and so run the best countermeasure to protect the network. According to the results, it is also shown that the Complete strategy, which includes the three DQN variations analyzed, is the one that allows obtaining agents with the best decision making to respond to attacks.