COVITON : Consistency driven integration of TPS and flow for virtual tryon

Abstract

Achieving realistic garment transfer while preserving both human and garment details, remains a challenging task in signal processing. The garment warping stage in virtual tryon plays a pivotal role in determining the visual fidelity of the final result. Existing methods commonly address this challenge by employing geometric transformations with control points using Thin-Plate Spline (TPS) or flow-based warping techniques. In this paper, we present an approach that jointly refines TPS and flow module’s tryon results utilizing a novel consistency constraint and fuses them through the integration of a (GFAM) Garment Fusion Attention Module. GFAM refines the flow warped garment using an attention based blending strategy that derives from TPS warped garment and integrates background person with the garment image to produce final tryon image. This not only preserves local and global textures but also achieves accurate garment deformation based on the target person’s pose. Our key innovation lies in the introduction of a novel Intra-Field Consistency loss, which ensures that the offset values computed by TPS and flow-based warping methods closely align with each other and GFAM block which facilitates a seamless garment and person fusion for realistic tryon generation. Importantly, our proposed framework represents the first attempt to manoeuvre the TPS module in a parser-free setting. Through extensive experiments and evaluations on VITON and VITON-HD datasets, we demonstrate the effectiveness of our method in achieving realistic and visually appealing state-of-the-art virtual tryon results.