An intelligent Q-learning-based tree routing method in underwater acoustic sensor networks

Abstract

Nowadays, underwater acoustic sensor networks (UASNs) have emerged as an advanced and promising technology for developing various underwater applications. However, several routing protocols have been suggested for these networks in recent years. This subject is still facing many challenges such as low propagation speed, low bandwidth, and energy restrictions. To solve the above-mentioned challenges, this paper proposes an intelligent Q-learning-based tree routing method called QTRU for underwater acoustic sensor networks. The proposed scheme includes a network bootstrapping process to be aware of local network topology and calculate the neighboring table related to each node. QTRU also contains a Q-learning-based tree construction process to transmit data from sensor nodes to the sink node. In the routing tree construction process, the reward function in the Q-learning algorithm consists of four parameters, including the depth of the node, remaining energy, successful transmission probability, and the size of the candidate set. In addition, to calculate the state set in the Q-learning algorithm, each node carries out two screening operations on its neighboring nodes on the network. The first screening operation ensure that each node in the routing tree has the least number of hops to the sink node. The second screening operation is to avoid the formation of routing loops between sensor nodes and ensure a tree-based network topology. QTRU also designs a recovery mechanism and allows the sensor nodes present in the void area to select their best parent node in the routing tree. Finally, QTRU is implemented in network simulator version 2 (NS2), and its results are compared with three routing methods, namely reinforcement learning-based opportunistic routing protocol (RLOR), Q-learning-based multi-level routing protocol (MURAO), and energy-efficient depth-based routing protocol (EE-DBR). These results show that QTRU improves the packet delivery rate (about 8.94%), data integrity (about 5.95%), delay (about 7.31%), energy consumption (about 9.74%), and the number of hops in the communication route (about 5.03%).