Resource Optimization for Log Shipper and Preprocessing Pipeline in a Large-Scale Logging System

In resource management, resource optimization is a usual technique to proceed for most professional organizations in order to reduce expenses and to dispose unnecessary resource usages. The European Organization for Nuclear Research (CERN) intends to implement a logging system based on AI for A Large Ion Collider Experiment detector, or ALICE. This system has been being implemented by using the Elasticsearch, Kibana, Beats, and Logstash also called ELK Stack which gives us the capability for the logs aggregation from systems and applications. Log data are collected from involved servers at CERN called First Level Processors (FLPs) nodes by Beats. These nodes run a large number of services when tasks are executed and generate a large volume of log data. Filebeat is used as a log shipper to transfer the data to Logstash, a server-side preprocessing pipeline. When Filebeat and Logstash are working together, there are many configurable factors affecting their efficiency. We then apply a factorial experiment to identify the significant factors and their correlation. These parameters are also optimized to find the best possible values of their configurations. Then, the resource usage can be minimized while a suitable performance of the system is maintained. The results of this study show that we can increase the efficiency of the system thanks to the adjusted values of the parameters. This can be used as a guideline for tuning some configurable parameters to optimize resource usage when there is a large amount of log data to be handled.