A disk based stream oriented approach for storing big data

Abstract

This paper proposes an extension to the generally accepted definition of Big Data and from this extended definition proposes a specialized database design for storing high throughput data from low-latency sources. It discusses the challenges a financial company faces with regards to processing and storing data and how existing database technologies are unsuitable for this niche task. A prototype database called CakeDB is built using a stream oriented, disk based storage design and insert throughput tests are conducted to demonstrate how effectively such a design would handle high throughput data as per the use case.