A Web of Things approach for learning on the Edge–Cloud Continuum

Internet of Things (IoT) devices provide constant, contextual data that can be leveraged to automatically reconfigure and optimize smart environments. Artificial Intelligence (AI) and deep learning techniques are tools of increasing importance for this, as Deep Reinforcement Learning (DRL) can provide a general solution to this problem. However, the heterogeneity of scenarios in which DRL models may be deployed is vast, making the design of universal plug-and-play models extremely difficult. Moreover, the real deployment of DRL models on the Edge, and in the IoT in particular, is limited by two factors: firstly, the computational complexity of the training procedure, and secondly, the need for a relatively long exploration phase, during which the agent proceeds by trial and error. A natural solution to both these issues is to use simulated environments by creating a Digital Twin (DT) of the environment, which can replicate physical entities in the digital domain, providing a standardized interface to the application layer. DTs allow for simulation and testing of models and services in a simulated environment, which may be hosted on more powerful Cloud servers without the need to exchange all the data generated by the real devices. In this paper, we present a novel architecture based on the emerging Web of Things (WoT) standard, which provides a DT of a smart environment and applies DRL techniques on real time data. We discuss the theoretical properties of DRL training using DTs, showcasing our system in an existing real deployment, comparing its performance with a legacy system. Our findings show that the implementation of a DT, specifically for DRL models, allows for faster convergence and finer tuning, as well as reducing the computational and communication demands on the Edge network. The use of multiple DTs with different complexities and data requirements can also help accelerate the training, progressing by steps.