Contextual Deep Reinforcement Learning for Flow and Energy Management in Wireless Sensor and IoT Networks

Efficient slot allocation and transmit-sleep scheduling is an effective access control mechanism for improving communication performance and network lifetime in resource-constrained wireless networks. In this paper, a decentralized and multi-tier framework is presented for joint slot allocation and transmit-sleep scheduling in wireless network nodes with thin energy budget. The key learning objectives of this architecture are: collision-free transmission scheduling, reducing energy consumption, and improving network performance. This is achieved using a cooperative and decentralized learning behavior of multiple Reinforcement Learning (RL) agents. The resulting architecture provides throughput-sustainable support for data flows while minimizing energy expenditure and sleep-induced packet losses. To achieve this, a concept of Context is introduced to the RL framework in order to capture network traffic dynamics. The resulting Contextual Deep Q-Learning (CDQL) model makes the system adaptive to dynamic and heterogeneous network load. It also improves energy efficiency when compared with the traditional tabular Q-learning-based approaches. The results demonstrate how this framework can be used for prioritizing application-specific requirements, namely, energy saving and communication reliability. The trade-offs among packet drop, energy expenditure, and learning convergence are studied, and an application-specific solution is proposed for managing them. The performance is compared against an existing state-of-the-art scheduling approach. Moreover, an analytical model of the system dynamics is developed and validated using simulation for arbitrary mesh topologies and traffic patterns.