Local programming language barriers in stream-based systems

Abstract

Stream-based data processing systems, such as Complex Event Processing or data stream mining systems, may be composed of several components which may be implemented in various programming languages. In distributed scenarios, computer networks typically represent important bottlenecks. However, the performance of data exchange in local contexts may be as important as the performance of data exchange via computer networks. Local programming language barriers may represent important bottlenecks for components that are located on the same computer system. In distributed scenarios, it may be beneficial to relocate components on a single physical host for exploiting the higher local data throughput. The properties of stream-based systems pose challenges like high throughput requirements but also open up optimization potential such as leveraging batched transfers. We performed an experimental analysis of ways for bridging local programming language barriers using the examples of C, Java, and Python and analyzed the impact of batched forwarding. While local data exchange can be expected to offer a higher throughput than exchange across networks, our results show that batch forwarding can increase the local throughput by factors of up to 47.6 and we measured net throughputs up to 39.5 Gbps.