Using Gossip Enabled Distributed Circuit Breaking for Improving Resiliency of Distributed Systems

Abstract

Distributed systems are rife with failures. Several resiliency patterns are used to improve the ability of these systems to tolerate faults and maintain functionality. The circuit breaker pattern is a popular resiliency pattern that is especially suitable for the case when the faults causing dependency failures take a variable amount of time to resolve. In this paper, we propose a modification to the traditional circuit breaker pattern. We also propose a gossip-based information dissemination protocol that enables the (modified) circuit breakers deployed on multiple client-service instances to take a concerted and more informed decision when a common dependency is facing persistent failures. We formally model the client-server systems that use traditional and the proposed distributed circuit breaker patterns in UPPAAL to analyze and compare their performance. The statistical model checking queries performed on the models show that, as compared to the traditional circuit breaker pattern, the distributed version results in fewer unsuccessful requests, that consume system/network resources, with practically the same total execution time under various availability conditions - only at the famously low cost of a robust gossip-based information dissemination protocol.