Deep Reinforcement Learning Based Transmission Scheduling for Sensing Aware Control

Abstract

Massive field data is wirelessly transmitted to the edge side to facilitate sensing and control in the emerging Industrial Internet of Things (IIoT) systems. Under the expanding transmission scheduling space and dynamic network conditions, balancing control performance and limited transmission resources is a fundamental challenge. For this problem, we propose a novel deep reinforcement learning (DRL)-based transmission scheduling method (DTSM), where sensing performance guarantee is introduced for its criticality in ensuring complete system observation and effective control. Specifically, taking system observability as the key metric, the time slots for multi-sensor data transmission under different control demands are properly reserved with theoretically guaranteed performance. Then, the primal-dual DRL framework is adopted to further improve the overall performance of system control and resource utilization by dynamically scheduling the transmission number of each sensor. The scheduling is based on the real-time states of sensing and wireless network, and the action space is determined according to our reserved time slots. Besides, after primal-dual updates, the scheduling results can satisfy the estimation error-evaluated constraint imposed for the ultimate control effect. Finally, the proposed method is applied to the industrial laminar cooling process and its effectiveness is fully demonstrated. Note to Practitioners—This paper is motivated by the requirement of balancing control performance and scarce transmission resources in industrial automation fields such as steel manufacturing, where massive sensor data is transmitted to the edge side through wireless networks. The expanding transmission scheduling space and dynamic network conditions have led to increased interest in advanced deep reinforcement learning (DRL) methods. However, few previous works have explored the impact of control demands on intelligent transmission scheduling design. For these issues, we propose a novel DRL-based transmission scheduling method (DTSM), where the time slots for multi-sensor data transmission are delicately reserved according to different control demands and dynamic scheduling is realized based on real-time states of sensing and wireless network. The overall performance of system control and resource utilization is improved, and practitioners can easily adjust method parameters to achieve the desired balance between the two aspects according to practical demands. Case studies in the industrial hot rolling process demonstrate the superiority of DTSM. Our future work will consider the joint scheduling of uplink-downlink transmissions and design the collaboration among multiple edge computing nodes (ECNs) to address the limitations of centralized learning methods. Besides, the proposed method can be extended to other industrial applications such as flight control system testing.