HEvaS: Headline Evaluation System

Automatic headline generation is a sub-task of oneline summarization with many reported applications. Evaluation of systems generating headlines is a very challenging and undeveloped area. In this paper, we introduce a system that performs automatic evaluation of systems in terms of a quality of the generated headlines. The evaluation is performed using multiple metrics for comparing evaluated headlines with the gold standard ones or measuring their coverage of main document topics. Both types of metrics evaluate headline's content and informativeness, but not grammatical structure. The only input required by our system is a set of documents with gold standard and automatically generated headlines. The Headline Evaluation System (HEvaS) provides a user with a choice from multiple (10) metrics, then calculates the chosen metrics, performs statistical analysis of the evaluated systems and visualizes the results. Multiple headline generation systems can be evaluated at the same run. This paper describes all evaluation metrics and architecture, utilized by our system. As an evaluation of the HEvaS, we perform a case study with a couple of baseline systems and report the results. Although we tested the system on English content only, the multilingual content can also be supported.