Automatic Web News Extraction Using Blocking Tag

Abstract

There are various approaches for web news extraction, including tree-edit distance approach that needs to assume the existence of web templates, visual wrapper based approach that requires large training sets and statistical approach whose flexibility is low. In this paper, a blocking tag based Web news extraction approach is proposed, which automatically detects the blocking tags that break the web page down into functional areas and then analyzes the web page according to the blocking tags to find out the news content. We have implemented the proposed news extraction approach in a news search engine which has been applied in business of an intelligence enterprise. Compared with related work, our approach does not require the web page templates or large training sets, and the complexity is lower.