MPEG NBMP testbed for evaluation of real-time distributed media processing workflows at scale

Abstract

Real-time Distributed Media Processing Workflows (DMPW) are popular for online media delivery. Combining distributed media sources and processing can reduce storage costs and increase flexibility. However, high request rates may result in unacceptable latency or even failures in incorrect configurations. Thus, testing DMPW deployments at scale is key, particularly for real-time cases. We propose the new MPEG Network Based Media Processing (NBMP) standard for this and present a testbed implementation that includes all the reference components. In addition, the testbed includes a set of configurable functions for load generation, monitoring, data-collection and visualization. The testbed is used to test Dynamic Adaptive HTTP streaming functions under different workloads in a standardized and reproducible manner. A total of 327 tests with different loads and Real-Time DMPW configurations were completed. The results provide insights in the performance, reliability and time-consistency of each configuration. Based on these tests, we selected the preferred cloud instance type, considering hypervisor options and different function implementation configurations. Further, we analyzed different processing tasks and options for distributed deployments on edge and centralized clouds. Last, a classifier was developed to detect if failures happen under a certain workload. Results also show that, normalized inter-experiment standard deviation of the metric means can be an indicator for unstable or incorrect configurations.