MapReduce based log file analysis for system threats and problem identification

Abstract

Log files are primary source of information for identifying the System threats and problems that occur in the System at any point of time. These threats and problem in the system can be identified by analyzing the log file and finding the patterns for possible suspicious behavior. The concern administrator can then be provided with appropriate alter or warning regarding these security threats and problems in the system, which are generated after the log files are analyzed. Based upon this alters or warnings the administrator can take appropriate actions. Many tools or approaches are available for this purpose, some are proprietary and some are open source. This paper presents a new approach which uses a MapReduce algorithm for the purpose of log file analysis, providing appropriate security alerts or warning. The results of this system can then be compared with the tools available.