Cross-corruption distillation for robust instruction following under text corruptions

Abstract

Traditional knowledge distillation transfers information from a large teacher model to a compact student by aligning static output distributions or internal representations for fixed inputs. Such formulations overlook how predictions change in response to input perturbations, even though robustness is critical for deployment. We interpret distillation as the transfer of functional response behavior under input variations and propose cross-corruption distillation (CCD), which aligns corruption-induced differences in teacher and student output via a Kullback–Leibler objective. Rather than matching absolute logits, CCD distills how the teacher redistributes its prediction mass under perturbations, transferring teacher-conditioned input sensitivity without architectural correspondence. On diverse multimodal benchmarks, CCD performs competitively, often outperforming conventional logit-based distillation. It also slows performance degradation as perturbation strength increases and remains stable across corruption strategies and training schedules. These results position CCD as a response-aware distillation framework integrating robustness directly into the distillation objective.