pytwanalysis: Twitter Data Management And Analysis at Scale

Abstract

Trends and communities in social media networks shape news cycles, politics, public governing, and economy these days. There is a wealth of information in the way users interact in the large social media networks, and state-of-the-art of mining network data from e.g. Twitter platform is limited by the narrow field of research or computing power. In this paper, we describe the new end-to-end Twitter network data management pipeline. We propose a scalable way to gather, store, and model rich relationships from Twitter networks. We also propose to analyze Twitter data using a combination of graph-clustering and topic modeling techniques at scale using multiple data science methods for graph construction and tweet data processing. We evaluate the proposed system on over 9 million tweets over five different Twitter datasets. We invite the community to add more features, as this end to end pipeline is released as an open source gitHub repository pytwanalysis [1], and as a python pip package pytwanalysis [2].