A Fuzzy Approach to Ranking Hyperlinks

Abstract

This paper presents a fuzzy approach to efficiently crawling topic related web pages using reinforcement learning and fuzzy clustering theory. The approach, FOA, takes the delayed reward into account and subsequently labels newly crawled web pages online. To minimize the bad effect of the naive Bayes classifiers, we adopt the fuzzy center-averaged clustering method to label crawled web pages, and using the calculated fuzzy memberships as class weights when calculating the fuzzy averaged Q values that map hyperlinks to future discounted rewards. The candidate hyperlinks are ranked according to their corresponding fuzzy averaged Q values, and the hyperlink with the optimal Q value is the best one to be crawled in the next step. Experiments of topic crawling tasks have shown FOA collects high harvest rate.