Challenges and opportunities for analysis based research in big data

Abstract

One response to the proliferation of massive datasets in many fields has been to develop ingenious ways to throw resources at the problem, for example, using massive fault tolerant storage architectures, supercomputing platforms, and parallel graph computation models. However, not all environments can support this scale of resources, and not all queries need an exact response. Massive and diverse operational datasets have been employed by large Internet Service Providers for a number of years, and mathematical methods have underpinned their response to the challenges of data scale, incompleteness and complexity that are prevalent both in ISP data and in big data more generally. This talk reviews some recent progress in this direction, and surveys some new roles for sampling methods in Big Data.