Inter-Service Communication among Microservices using Kafka Connect

Abstract

The new age software should be highly scalable and easily maintainable. The most used architecture to ensure this is the microservice architecture. The software is composed of independent small services in microservice architecture that communicate over well-defined APIs. The communication for the asynchronous flows cannot be done by HTTP APIs which are synchronous (request-reply pattern) as it leads to high consumption of the resources which in turn leads to increase in the traffic of both the microservices. In order to tackle this problem for asynchronous nature, a message queue like Kafka, RabbitMQs are used where a publisher - subscriber model is employed. The usage of this further leads to the bottlenecks in publisher when there are more threads writing to the publisher, but publisher has fixed number of threads to write to message broker partitions. This bottleneck leads to the decrease in the performance and increase in the response time. In this paper, we present the methodology and the pipeline implementation to decrease the bottleneck and improve the time taken for publish - subscribe model for the microservices and reduce the CPU and memory usage of the containers where the microservice is running.