Near Real-Time Tracking at Scale

Abstract

Clickstream data analysis involves collecting, analyzing and aggregating data for business analytics. Key business indicators such as user experience, product checkout flows, failed customer interactions are computed based on this data. A/B testing [18] or any data experimentation use clickstream data stream to compute business lifts or capture user feedback to new changes on the site. Handling such data at scale is extremely challenging, especially to design a system ensuring little to no data loss, bot filtering, event ordering, aggregation and sessionization of user visit. The entire operation must be near real-time so that computations performed can be fed back into services which can help in targeted personalization and better user experience. Sessions capture group of user interactions within stipulated time frame. Business metrics often computed on these user sessions. User sessions are therefore critical for business analytics as they represent true user behavior. We describe the process of creating a highly available data pipeline and computational model for user sessions at scale.