Imputation of missing pixel art character poses with differentiable palette quantization

Abstract

Designing pixel art character sprites with numerous animation frames is a labor-intensive process that often involves repetitive work. To streamline this task, we propose a method that automates sprite generation, allowing artists to focus on the creative aspects. Our work addresses the challenge of synthesizing a character sprite in a target pose using reference images from other viewing angles. We formulate this as a missing data imputation problem and introduce a generative adversarial network that reconstructs the desired pose from those already available among the back, left, front, and right directions. Unlike baseline models, our proposed generator utilizes all available poses of a character to enhance the quality of the generated image. Additionally, it does not introduce small variations of the same colors and instead produces images that strictly follow a predefined color palette. Crucially, our model ensures adherence to the palette by incorporating a novel operation of differentiable quantization of pixel values, making it suitable for end-to-end training. Compared to baseline models proposed for generating a new pose from a single one, our approach produces images with better FID (34.09% lower) and $L_1$ distance to the ground truth (22.66% lower). It also shows superior quality through visual inspection. Additionally, as the generator selects colors from a desired palette, similar to how human artists create pixel art, the generated images are more readily useful, eliminating the need for a post-processing step to restrict the color.