VarLenMARL: A Framework of Variable-Length Time-Step Multi-Agent Reinforcement Learning for Cooperative Charging in Sensor Networks

This paper studies cooperative charging, in which multiple mobile chargers cooperatively provide wireless charging services in a Wireless Rechargeable Sensor Network (WRSN). The ultimate goal of this cooperative charging is the long-term optimization that maximizes both the lifetime of all sensor nodes and the charging utility of each Mobile Charger (MC). We have attempted to apply Multi-Agent Reinforcement Learning (MARL) algorithms to this problem. Unfortunately, similar to existing methods, MARL algorithms also fail early in cooperative charging. We found that an MARL algorithm trained in each time-step of fixed length is neither accurate nor efficient in cooperative charging. We propose a new MARL framework, called VarLenMARL. For the accuracy of reward estimation, VarLenMARL allows each MC completes an action within a time-step of variable length before estimating rewards. Furthermore, we design a special mechanism in VarLenMARL for the long-term optimality of cooperative charging within a WRSN. Our results show that algorithms implemented on VarLenMARL achieved both higher charging utility of MCs and longer lifetime of sensor nodes.