Towards a benchmark for Web site extractors: a call for community participation

The purpose of this paper is to propose a benchmark for comparing fact extractors for Web sites and to invite interested researchers and practitioners to participate in its development. Fact extraction is a fundamental and difficult problem in both traditional software reverse engineering and Web site reverse engineering. In both domains, there are often irregularities in the input that violate an extractor's unstated assumptions. Consequently, it is difficult to predict how an extractor will perform in a given input. To remedy this problem, we created a benchmark for comparing fact extractors for the C++ programming language. We found that this benchmark improved our understanding of fact extraction, the tools produced, and the maturity of the community. The same approach, we believe, will be beneficial for Web site extractors and we propose WebETS (Web site Extractor Test Suite.) In this paper we give some starting points for the design of WebETS and ask others to join in the effort.