ObsCon: Integrated Monitoring and Control for Parallel, Real-Time Applications

Abstract

A large class of emerging compute-intensive applications demand real-time or near real-time processing guarantees on streaming data. Sensor processing in particular, has stringent latency requirements for carrying out its digital processing for rapidly incoming radar data streams. The consequent demands on the cluster middleware used to run such codes include (i) efficient online observation of current application performance, coupled with (ii) highly responsive controllers able to dynamically adjust the application's input-and data-dependent runtime behavior. We present the Obs(erver)Con(troller) software for online monitoring and control, which based on specifications of acceptable application states and tunable knobs within the execution environment, ensures that application performance falls within acceptable limits. ObsCon topologies are dynamic, making possible the runtime association of ObsCon methods with arbitrary DAG-structured, distributed/parallel stream processing applications running on high end cluster machines. This paper describes the ObsCon software and its 'grey box' use with a high performance cluster code that exports to ObsCon select 'hooks' for online monitoring and control -- Adaptive Digital Beamforming for a phase-array radar system.