Performance Evaluation of Density-Based Clustering Methods for Categorizing Web Robot Sessions

Abstract

Web servers are flooded with programmed web scripts (termed as web robots or web crawlers) generated HTTP requests. The detection of web traffic generated by automated web scripts at server end is essential for blocking or at least minimizing the impact on server resources and services. Web robot sessions are characterized by their own navigational behavior and extracted features. Session labeling may be used for identification of robots by supervised learning. However, due to high frequency and dynamically changing behavior of robots session labeling is not feasible all the time. Therefore, in the absence of any label with generated session’s unsupervised learning prove very useful for segregating human and robot sessions. In this paper, the effectiveness of different density-based algorithms is evaluated on user session data. The user sessions are clustered in four possible categories such as human sessions, tentative human and robot sessions and tentative robots. The experiments are performed on five labeled datasets of varying session length. Three most popular density-based algorithms such as density-based spatial clustering with noise (DBSCAN), ordering points to identify the clustering structure (OPTICS), and density-based clustering (DENCLUE) are used for session clustering. The comparative performances of used clustering algorithms are evaluated using supervised and unsupervised validation indexes including Rand, Jaccard, Silhouette, and Davis-Bouldin index.