On Domain Generalization for Batched Prediction: the Benefit of Contextual Adversarial Training

Abstract

This paper considers the domain generalization problem in which the labels of the query batch may be predicted at once and the model is allowed to assign a label to a query example utilizing all information contained in the query batch. In this setting, we identify two problems in the standard adaptive risk minimization (ARM) framework, namely, mismatched context and overfitted context, which are most severely manifested respectively at small and large support batch sizes. The existence of these problems signifies a fundamental tradeoff between within-domain learning and cross-domain generalization. We also propose two context adversarial training approaches to alleviate these problems so as to achieve a better tradeoff. We demonstrate experimentally that the proposed approaches uniformly outperform the standard ARM training for all choices of support batch sizes.