Effective data stream mining using ensemble on cloud with load balancing (E2CL)

Abstract

Data stream is generated everywhere with ever increasing speed. There is a need for efficient stream processing systems and optimal algorithms to mine all items of these streams to accurately predict the knowledge in limited time. In the existing approaches, there are some limitations like one-pass, sampling and load shedding on processing the streams which trade-off in accuracy. There are some approaches which use the distributed computing, grid computing and cloud computing technologies to deal with these challenges. This paper proposes a new approach to reduce the overhead of processing the already processed items. In this approach there will be a central system called model aggregator that will pull the learnt knowledge from all the stream processing systems, combine those knowledge and then will push to all the cloud processing systems in certain time interval. Having this combined knowledge, the participating stream processing systems' overhead is reduced that will increase the availability of the systems to handle the additional streams. Also since the cloud systems can be provisioned in advance or on-demand when the peak streaming occurs, the window dropping can be avoided.