Scalable and elastic realtime click stream analysis using StreamMine3G

Abstract

Click stream analysis is a common approach for analyzing customer behavior during the navigation through e-commerce or social network sites. Performing such an analysis in real-time opens up new business opportunities as well as increases revenues as recommendations can be generated on the fly making a previously unknown product to the potential customer attractive. As click streams are highly fluctuating as well as must be processed in real time, there is a high demand for Event-Stream-Processing (ESP) engines that are (1) horizontally as well as vertically scalable, (2) elastic in order to cope with the fluctuation in the data stream, and (3) provide efficient state management mechanisms in order to drive such kind of analysis. However, the majority of the nowadays ESP engines such as Apache S4 or Storm provide neither explicit state management nor techniques for elastic scaling. In this paper, we present StreamMine3G, a scalable and elastic ESP engine which provides state management out of the box, scales with the number of nodes as well as cores and improves performance due to a novel delegation mechanisms lowering contention on state as well as network links caused by fluctuations and temporary imbalances in the data streams.