ADLS Gen 2 for web server log data analysis

Abstract

Before the advent of Big Data, there were few possibilities for processing TB of data sets or more. As data generation capacity has grown, so has the need to store and process large volumes of data. The web server log files contain important and valuable information about events related to customer activities in the online environment, server activities and its response to customer requests, and much more. It is well known that web server log files are files that contain large volumes of data and grow very quickly in size, servers having to delete them once they reach a certain size. Thus, many valuable data are lost, data on which various analyzes can be made in order to obtain valuable information that will later lead to improvements on several levels. It is important to find cost-effective and easily accessible techniques for storing and analyzing these files, which are also considered to be of a high privacy level holding information classified as personal data. The main objective is the research for the development and implementation of an innovative solution for storing and analyzing large volumes of web server log files using cloud storage - ADLS Gen2.