A stream-based architecture for the management and on-line analysis of unbounded amounts of simulation data

Conducting simulation studies can mean to execute a multitude of parameter configurations, for each of these we may need to execute a vast number of replications, and each single replication may mean the need to process a significant amount of data. Here, we propose a stream-based architecture that aligns data processing and buffering with the actual data usage during simulation to make the most of available memory. This turns away from the first-write-then-read approach, often utilizing databases or plain files as temporary storage. Instead, data are processed on the fly. By introducing a processing graph, which distinguishes between buffering and processing nodes, a flexible analysis of simulation data is achieved. As the data are processed close to their generation, the developed architecture fits well to a distributed execution of simulation studies. We illustrate how the stream-based architecture integrates into simulation workflows.