Decentralization of Control Loop for Self-Adaptive Software through Reinforcement Learning

In a decentralized self-adaptive software, multiple control loops provide self-adaptation capabilities to the components these manage. For this, these control loops need to coordinate to continuously satisfy some local QoS goals of each managed component and global QoS goals concerning the whole system in a changing environment. This is accomplished by choosing variants of the managed system (component) that satisfy these goals. As goal conformance requires coordination, a control loop requires choosing the variant that leads to maximum goal conformance considering the variant selection strategies by other control loops. An overall goal conformance calculation mechanism is also needed that captures the local and global goal violations. This paper proposes a decentralized reinforcement learning-based self-adaptation technique considering these issues. A reinforcement learning technique, Q-learning helps to learn the maximum achievable goal conformance choosing a specific variant. The other control loop strategies are estimated by observing their variant selection and incorporated with Q-learning for better variant selection. An overall goal conformance calculation technique is also proposed that dynamically adjusts weights on the local and global goals to emphasize violated goals. The proposed approach was evaluated using a service-based Tele Assistance System. It was compared with two approaches - random variant selection and variant selection ignoring other control loop strategies. The proposed technique outperformed both with maximum overall goal conformance. The proposed dynamic weight update mechanism was compared with a static weight-based one. The dynamic technique outperformed the static one continuously satisfying all the goals.