Utilizing Reinforcement Learning to Autonomously Mange Buffers in a Delay Tolerant Network Node

In order to effectively communicate with Earth from deep space there is a need for network automation similar to that of the Internet. The existing automated network protocols, such as TCP and IP, cannot work in deep space due to the assumptions under which they were designed. Specifically, protocols assume the existence of an end-to-end path between the source and destination for the entirety of a communication session and the path being traversable in a negligible amount of time. In contrast, a Delay Tolerant Network is a set of protocols that allows networking in environments where links suffer from high-delay or disruptions (e.g. Deep Space). These protocols rely on different assumptions such as time synchronization and suitable memory allocation. In this paper, we consider the problem of autonomously avoiding memory overflows in a Delay Tolerant Node. To that end, we propose using Reinforcement Learning to automate buffer management given that we can easily measure the relative rates of data coming in and out of the DTN node. In the case of detecting overflow, we let the autonomous agent choose between three actions: slowing down the client, requesting more resources from the Deep Space Network, or selectively dropping packets once the buffer nears capacity. Furthermore, we show that all of these actions can be realistically implemented in real-life operations given current and planned capabilities of Delay Tolerant Networking and the Deep Space Network. Similarly, we also show that using Reinforcement Learning for this problem is well suited to this application due to the number of possible states and variables, as well as the fact that large distances between deep space spacecraft and Earth prevent human-in-the-loop intervention.