Capturing user access patterns in the Web for data mining

Abstract

Existing methods for knowledge discovery in the Web are mostly server-oriented and approaches taken are affected by the use of proxy servers. As a result, it is difficult to capture individual Web user behavior from the current log mechanism. As an effort to remedy this problem, we develop methods for design and implementation of an access pattern collection server to conduct data mining in the Web. We also devise an innovative method, called page conversion, which converts the original Web pages to enciphered ones so that the devised data collection mechanism will not be deliberately bypassed. With the concept of page conversion, the methods we proposed involves a mechanism of software downloading to resolve the difficulty imposed by proxy servers and to effectively capture the Web user behavior. Using the devised mechanism, traversal patterns are generated and compared to those produced by the ordinary Web servers to validate our results. It is shown that the traversal patterns resulting from the devised system are not only more informative but also more accurate than those generated by ordinary Web servers, showing the importance and the usefulness of the mechanism devised.