Verifying Zookeeper based on Model-Based runtime Trace-Checking using TLA+

Zookeeper atomic broadcast (Zab) is a atomic broadcast protocol specially designed for the distributed coordination service Zookeeper to support rout recovery. It is widely used in such scenarios as cluster management, load balancing and data publish and subscription. It is the core algorithm for ensuring Zookeeper data consistency. In this article, we design a Model-Based runtime Tracechecking system based on formal specification language TLA+ and its model checking tool TLC and use it to perform online verification against the behaviors of distributed Zookeeper cluster according to trace, which aims to monitor and check whether the actual execution of Zookeeper cluster complies with the Zab protocol specification. Firstly, stubs are added to relevant Zookeeper code modules which are synchronized with the implementation of Zab protocol state transitions to guarantee specific trace info generation and output to the target log file. Secondly, a formal model is built with TLA+ which can analyzes and specifies the state sequences info of Zookeeper cluster provided by trace, and the correctness and safety properties for this specification are also defined according to Zab protocol. A series of operators were also developed for TLC to load the trace info in history with different formats from the log files for model checking. The final experiment shows that this checking system can provide light-weighted and feasible formal method for verifying actual execution behaviors of distributed systems.