Self-training by Reinforcement Learning for Full-autonomous Drones of the Future*

Abstract

Drones are rapidly increasing their activity in the airspace worldwide. This expected growth of the number of drones makes human-based traffic management prohibitive. Avionics systems able to sense-and-avoid obstacles and specially visual flight rules (VFR) traffic are under research. Moreover, to overcome loss-link contingencies, drones have to be able to act autonomously. In this paper we present a drone concept with a full level of autonomy based on Deep Reinforcement Learning (DRL). From the first flight until the accomplishment of its final mission, the drone has no need for a pilot. The only human intervention is the engineer programming the artificial intelligence algorithm used to train and then to control the drone. In this paper we present the preliminary results for an environment which is a realistic flight simulator, and an agent that is a quad-copter drone able to execute 3 actions. The inputs of the agent are the current state and the accumulated reward. Experiments include self-learning periods up to 3 days, followed by one hundred full-autonomous flight tests. Three different DRL algorithms were used to obtain the training models, based in Q-learning reinforcement learning. Results are very promising, with around an 80 percent of test flights reaching the target. In comparison with the results of a human pilot, acting in the same simulated environment and using the same three actions, the DRL methods demonstrated unequal results, depending on the learning algorithm used. We applied some enhancements in the training, with the creation of checkpoints of the training model every time a better solution is found. In a near future we expect to achieve results similar to the performance of a human pilot to support the idea of full-autonomous drones through DRL methods.