High speed streaming data analysis of web generated log streams

Abstract

Web logs provide useful insight of large scale web based applications and helpful in deriving web usage patterns. Since, web usage patterns are available at a high rate and a high volume and also continuously updating in a real time environment, must be handled through modern big data architectures supported by powerful real time big data processing tools. Web generated log streams have most significant impact when it is feasible to analyze them at a time when they are emitted. In proposed research work, an advanced stream analytics framework especially for web generated log streams has been proposed by using the dataset of web access logs representing HTTP requests received by NASA Kennedy Space Center Server. The proposed framework can resourcefully handle the challenging issues associated to manage multiple web based log streams that are distributed across a fleet of web based applications and present a summarized view of statistical profile of web based applications which may be useful for web usage mining.