A Distributed Message Queuing Mechanism for a Mailing System with High Performance and High Availability

Abstract

Today, informing users, customers, members through e-mail, text message or through any other mean is crucial for companies. Information methods get more critical if user/message volume gets higher and message delivery must be on time. Because of performance reasons, the designed systems must be high-available and distributed. In this paper, we discuss how we built a high-performance mailing system by using Apache Kafka as an internal sub-system messaging and how we replaced the existing central database-oriented queuing architecture with this new system. Also, we discuss how we overcame the performance problems of the previous design with new queueing mechanism. About 40 million daily mails are passed through the pipeline successfully after we built the new architecture.