LambdaLink: an Operation Management Platform for Multi-Cloud Environments

Abstract

The last several years have seen an unprecedented growth in data availability, with dynamic data streams from sources ranging from social networks to small, inexpensive sensing devices. This new data availability creates an opportunity, especially in geospatial data science where this new, dynamic, data allows novel insight into phenomena ranging from environmental to social sciences. Much work has focused on creating venues or portals for publishing and accessing such dynamic datasets. However access to data in itself is not sufficient to turn data into information the data needs to be filtered, correlated, and otherwise analyzed using methods that are dynamically developed and constantly improved by a distributed community of experts. Further, these methods are increasingly required to deliver results with specific qualities of service, e.g., providing results by a certain deadline or ensuring a certain accuracy of the results. Delivering such qualities of service requires generic but often sophisticated tools managing the execution of operations and ensuring their correctness. This paper presents LambdaLink, an operation management platform for multi-cloud environments, and explains how it supports the structured contribution and repeatable, time-controlled execution of operations. We describe the architecture and implementation of LambdaLink, its approach to appliance management in a multi-cloud context, and compare it with related systems.