CIMLA: Checkpoint Interval Message Logging Algorithm in Kafka Pipeline Architecture

In big data era, the reliability of real-time messages is the most important point for many business needs. Real-time messaging system establishes to handle real-time requirements and reliability requirements in business applications. Apache Kafka is a highly scalable publish-subscribe messaging system. The effect of reliability in Apache Kafka impact on the different use cases. Fault tolerance becomes the main factor to improve the reliability of Apache Kafka. Apache Kafka needs to handle the challenges of the message loss rate in server failure. To improve the message recovery, Kafka requires to manage the message delivery process. This paper introduces CIMLA (Checkpoint Interval Message Logging Algorithm) to enhance the message delivery in Apache Kafka. The system evaluates the message loss rate, throughput and latency in the server failure case. The system achieves the message recovery process by using CIMLA which effectively controls the lost messages with identification. The experimental results emphasize the improvement of message recovery and verify the reliability of message delivery in Apache Kafka.