Analysing Log Files For Web Intrusion Investigation Using Hadoop

Abstract

The process of analyzing large amount of data from the log file helps organization to identify the web intruders' activities as well as the vulnerabilities of the website. However, analyzing them is totally a great challenge as the process is time consuming and sometimes can be inefficient. Existing or traditional log analyzers may not able to analyze such big chunk of data. Therefore, the aim of this research is to produce an analysis result for web intrusion investigation in Big Data environment. In this study, web log was analyzed based on attacks that are captured through web server log files. The web log was cleaned and refined through a log-preprocessing program before it was analyzed. An experimental simulation was conducted using Hadoop framework to produce the required analysis results. The results of this experimental simulation indicate that Hadoop application is able to produce analysis results from large size web log files in order to assist the web intrusion investigation. Besides that, the execution time performance analysis shows that the total execution time will not increase linearly with the size of the data. This study also provides solution on visualizing the analysis result using Power View and Hive.