Next Generation of Impersonator Bots: Mimicking Human Browsing on Previously Unvisited Sites

The development of Web bots capable of exhibiting human-like browsing behavior has long been the goal of practitioners on both side of security spectrum - malicious hackers as well as security defenders. For malicious hackers such bots are an effective vehicle for bypassing various layers of system/network protection or for obstructing the operation of Intrusion Detection Systems (IDSs). For security defenders, the use of human-like behaving bots is shown to be of great importance in the process of system/network provisioning and testing. In the past, there have been many attempts at developing accurate models of human-like browsing behavior. However, most of these attempts/models suffer from one of following drawbacks: they either require that some previous history of actual human browsing on the target web-site be available (which often is not the case), or, they assume that 'think times' and 'page popularities' follow the well-known Poisson and Zipf distribution (an old hypothesis that does not hold well in the modern-day WWW). To our knowledge, our work is the first attempt at developing a model of human-like browsing behavior that requires no prior knowledge or assumption about human behavior on the target site. The model is founded on a more general theory that defines human behavior as an 'interest-driven' process. The preliminary simulation results are very encouraging - web bots built using our model are capable of mimicking real human browsing behavior 1000-fold better compared to bots that deploy random crawling strategy.