MonEx: An Integrated Experiment Monitoring Framework Standing on Off-The-Shelf Components

Abstract

Most computer experiments include a phase where metrics are gathered from and about various kinds of resources. This phase is often done via manual, non-reproducible and error-prone steps. Infrastructure monitoring tools facilitate collecting experiments' data to some extent. However, there is no conventional way for doing so, and there is still much work to be done (e.g. capturing user experiments) to leverage the monitoring activity for monitoring experiments. To overcome those challenges, we define the requirements of experiments monitoring, clarifying Experiment Monitoring Frameworks' scope and mainly focusing on reusability of experiments' data, and portability of experiments' metrics. We then propose MonEx EMF that satisfies those requirements. MonEx is built on top of infrastructure monitoring solutions and supports various monitoring approaches. It fully integrates into the experiment workflow by encompassing all steps from data acquisition to producing publishable figures. Hence, MonEx represents a first step towards unifying methods of collecting experiments' data.