Towards self-managing cloud-scale computing platforms: Experiences and challenges

Abstract

Summary form only given, as follows. The complete presentation was not made available for publication as part of the conference proceedings. More and more companies heavily rely on massive data analysis of many kinds to understand data insights and drive business decisions. To support this ever-increasing need, big data computing platforms have grown to an unprecedented scale, way beyond human manageability. In this talk, I'll share our experiences at Alibaba to enable our big data platforms to configure, optimize, monitor, and protect themselves automatically, including automatic version testing and deployment control, system health monitoring and alert, automatic physical design/data placement/storage optimization, etc. I'll also outline some outstanding research and engineering challenges.