AoI-aware transmission control in real-time mmwave energy harvesting systems: a risk-sensitive reinforcement learning approach

The evolution of enabling technologies in wireless communications has paved the way for supporting novel applications with more demanding QoS requirements, but at the cost of increasing the complexity of optimizing the digital communication chain. In particular, Millimeter Wave (mmWave) communications provide an abundance of bandwidth, and energy harvesting supplies the network with a continual source of energy to facilitate self-sustainability; however, harnessing these technologies is challenging due to the stochastic dynamics of the mmWave channel as well as the random sporadic nature of the harvested energy. In this paper, we aim at the dynamic optimization of update transmissions in mmWave energy harvesting systems in terms of Age of Information (AoI). AoI has recently been introduced to quantify information freshness and is a more stringent QoS metric compared to conventional delay and throughput. However, most prior art has only addressed average-based AoI metrics, which can be insufficient to capture the occurrence of rare but high-impact freshness violation events in time-critical scenarios. We formulate a control problem that aims to minimize the long-term entropic risk measure of AoI samples by configuring the “sense & transmit” of updates. Due to the high complexity of the exponential cost function, we reformulate the problem with an approximated mean-variance risk measure as the new objective. Under unknown system statistics, we propose a two-timescale model-free risk-sensitive reinforcement learning algorithm to compute a control policy that adapts to the trio of channel, energy, and AoI states. We evaluate the efficiency of the proposed scheme through extensive simulations.