Big data in [keynote speaker 2]

Abstract

Summary form only given, as follows. The resolution power of experiments is improving steadily and the data rate production is rapidly increasing. The success of the experiments depends critically on handling effectively and efficiently huge amounts of data. The on-detector reduction of the data rate will be a major topic as only a fraction of the data can be archived for later long-term analyses. In the project “Large Scale Data Management and Analysis” (LSDMA) several Helmholtz centres and German universities are cooperating in order to support researchers in maintaining their huge amounts of data. Besides supporting individual scientific communities, generic services are being developed, e.g. Federated identity management; Federated data access; Meta data repositories; Archive services; Monitoring, modelling, optimization; and Data intensive computing & analysis. The talk will explore the general challenges of Big Data. Several instructive examples from different scientific communities are presented. An overview of the current status of the LSDMA project is given. In addition, recent results on real-time and near-real time analysis of Big Data are presented.