A Living Lab Architecture for Reproducible Shared Task Experimentation

Abstract

No existing evaluation infrastructure for shared tasks currently supports both reproducible onand offline experiments. In this work, we present an architecture that ties together both types of experiments with a focus on reproducibility. The readers are provided with a technical description of the infrastructure and details of how to contribute their own experiments to upcoming evaluation tasks.