Experiment-driven improvements in Human-in-the-loop Machine Learning Annotation via significance-based A/B testing

We present an end-to-end experimentation framework to improve the human annotation of data sets used in the training process of Machine Learning models. It covers the instrumentation of the annotation tool, the aggregation of metrics that highlight usage patterns and hypothesis-testing tools that enable the comparison of experimental groups, to decide whether improvements in the annotation process significantly impact the overall results. We show the potential of the protocol using two real-life annotation use cases.