Processing 6 billion CDRs/day: from research to production (experience report)

Abstract

A call detail record (CDR), is a data record produced by a telephone exchange or other telecommunications equipment documenting the details of a phone call that passed through the exchange or equipment. Telecommunications companies (or "telcos") use CDRs for purposes of billing, extracting business intelligence, fraud detection, etc. However, they face a Big Data challenge -- many telcos get billions of CDRs per day, and are unable to keep up with these data rates. In this paper, we describe a stream processing solution for processing CDRs that allows scaling the processing to handle 6 billion CDRs per day for a certain telco. We describe the stream processing application (running on the IBM InfoSphere Streams platform) that performs CDR mediation and analysis in real-time. We also describe various business and operational constraints and the legacy software ecosystem - seldom discussed in academic gatherings - that make the problem more challenging than originally meets the eye. The outcome of our work is a highly configurable and scalable CDR processing stream with several functional and performance capabilities that are a first for the telecommunication industry.