In-memory, high speed stream processing

To enable load prediction we require three ingredients; Data acquisition, Accurate prediction algorithms and timely prediction. This work is focused on the third ingredient i.e. Timely prediction. Timely prediction is required for proper functioning of smart grid. Towards this we offer a scalable, distributed and incremental solution. We present a solution that utilise multiple techniques to evade performance degradation maintaining timing requirements. These techniques include:(1) Splitting the nested processing into stages to maintain high throughput. (2) Customised median finding algorithms to generate timely output from high volume data. (3) Pre-fetching to eliminate disk access time for accessing the historical data. Implementing these together, our technique is able to perform 11K predictions per sec, and 2.5K outlier computations per sec.