Distributed and Dynamic Clustering For News Events

Proceedings of the 12th ACM International Conference on Distributed and Event-based Systems Pub Date : 2018-06-25 DOI:10.1145/3210284.3219774

Vinay Setty

{"title":"Distributed and Dynamic Clustering For News Events","authors":"Vinay Setty","doi":"10.1145/3210284.3219774","DOIUrl":null,"url":null,"abstract":"The primary consumption of news is now increasingly online and has resulted in a large volume of online news from varied news outlets. Consequently news aggregators have become popular for clustering, ranking and personalization of news which process millions of news articles each day. In addition, since news articles stream constantly, there is a need for a scalable event-based system which can facilitate news mining in an online fashion. To address these challenges, we propose a distributed framework to process news articles and cluster them to facilitate many news mining tasks. The core of our system is a novel and scalable distributed clustering algorithm using Locality Sensitive Hashing which is robust to outliers and noise. In addition, we also propose an online version of the clustering algorithm to dynamically maintain the news event clusters. We implement the proposed solution on Apache Spark. Using a large news collection with over 8 million news articles, we show that our approach outperforms widely-used clustering techniques such as K-Means both in run time and clustering quality.","PeriodicalId":412438,"journal":{"name":"Proceedings of the 12th ACM International Conference on Distributed and Event-based Systems","volume":"11 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2018-06-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Proceedings of the 12th ACM International Conference on Distributed and Event-based Systems","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1145/3210284.3219774","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 0

Abstract

The primary consumption of news is now increasingly online and has resulted in a large volume of online news from varied news outlets. Consequently news aggregators have become popular for clustering, ranking and personalization of news which process millions of news articles each day. In addition, since news articles stream constantly, there is a need for a scalable event-based system which can facilitate news mining in an online fashion. To address these challenges, we propose a distributed framework to process news articles and cluster them to facilitate many news mining tasks. The core of our system is a novel and scalable distributed clustering algorithm using Locality Sensitive Hashing which is robust to outliers and noise. In addition, we also propose an online version of the clustering algorithm to dynamically maintain the news event clusters. We implement the proposed solution on Apache Spark. Using a large news collection with over 8 million news articles, we show that our approach outperforms widely-used clustering techniques such as K-Means both in run time and clustering quality.

查看原文本刊更多论文

新闻事件的分布式和动态聚类

新闻的主要消费现在越来越多地在网上，这导致了来自各种新闻媒体的大量在线新闻。因此，新闻聚合器已经成为流行的聚类，排名和个性化的新闻，每天处理数以百万计的新闻文章。此外，由于新闻文章流不断，因此需要一个可扩展的基于事件的系统，以在线方式促进新闻挖掘。为了解决这些挑战，我们提出了一个分布式框架来处理新闻文章，并将它们聚类，以促进许多新闻挖掘任务。该系统的核心是一种新颖的、可扩展的分布式聚类算法，该算法使用局域敏感散列，对异常值和噪声具有鲁棒性。此外，我们还提出了一种在线版本的聚类算法来动态维护新闻事件聚类。我们在Apache Spark上实现了该解决方案。使用超过800万篇新闻文章的大型新闻集合，我们表明我们的方法在运行时间和聚类质量上都优于广泛使用的聚类技术，如K-Means。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Proceedings of the 12th ACM International Conference on Distributed and Event-based Systems

自引率

0.00%

发文量