DRIM: MDL-Based Approach for Fast Diverse Summarization

Automated text summarization extracts essential information from original text and presents it in a predefined number of words. In this paper, we introduce an unsupervised extractive summarization approach that takes its roots from the SLIM dataset compression algorithm [1] based on the Minimum Description Length (MDL) principle [2], [3]. Our approach represents text as a transactional dataset, where sentences are transactions and normalized words are items. We use the SLIM algorithm (SLIM is not an abbreviation, it is Dutch word for 'smart') to solve the main bottleneck of the MDL computation, which is the generation of all frequent itemsets as a first step of the model construction. Additionally, we add a diversity constraint to the model in order to decrease appearance of repeated information in a summary. We introduce DRIM (Diversed SLIM) algorithm that performs unsupervised summarization, both generic and query-based, and does not require parameter tuning. We evaluate our summarizer on texts in English, but it can be easily extended to other languages.