Blue Danube: A Large-Scale, End-to-End Synchronous, Distributed Data Stream Processing Architecture for Time-Sensitive Applications

An extensive list of time-sensitive applications requiring ultra-low latency ranging from a few microseconds to a few milliseconds are presented in recent publications and IEEE standards. Time-sensitive applications, include industrial, critical healthcare and transportation applications as also applications for Smart Grids and the Internet of Vehicles – one of the most active research fields of Intelligent Transportation Systems of Smart Cities. In this work, we mainly set our focus on the suite of safety applications which attracts strong interest from the research community, as it aims to avoid road accidents and save lives. The IEEE Time-Sensitive Networking (TSN) set of standards specifies fundamental real-time characteristics. Nevertheless, as TSN works on Data Link layer (Layer 2 of the OSI model) the benefits of these characteristics fade away when other layers are crossed from the Application layer (Layer 7). Indicatively, recent research works report latencies on the order of tens of seconds when benchmarking Data Stream Processing and IoT platforms, and thus they are not suited for time-critical applications. Such platforms mainly use loosely coupled components with asynchronous communication. On Application layer, we propose a novel End-to-End Synchronous, Distributed Architecture for Large-Scale, High-Bandwidth, Ultra-Low Latency Data Stream Processing. Through our Big Data Stream analysis experiments (4.7 Gbit/s total average aggregated throughput, 1 Terabyte in-memory distributed database, 4 milliseconds average query latency) we have demonstrated the suitability of our architecture for time-sensitive applications such as accident avoidance for the Internet of Vehicles.