KeyBoxGAN: enhancing 2D object detection through annotated and editable image synthesis

Abstract

Sample augmentation, especially sample generation is conducive for addressing the challenge of training robust image and video object detection models based on the deep learning. Still, the existing methods lack sample editing capability and suffer from annotation work. This paper proposes an image sample generation method based on key box points detection and Generative adversarial network (GAN), named as KeyBoxGAN, to make image sample generation labeled and editable. KeyBoxGAN firstly predefines key box points positions, embeddings which control the objects’ positions and then the corresponding masks are generated according to Mahalanobis–Gaussuan heatmaps and Swin Transformer-SPADE generator to control objects’ generation regions, as well as the background generation. This adaptive and precisely supervised image generation method disentangles object position and appearance, enables image editable and self-labeled abilities. The experiments show KeyBoxGAN surpasses DCGAN, StyleGAN2 and DDPM in objective assessments, including Inception Distance (FID), Inception Score (IS), and Multi-Scale Structural Similarity Index (MS-SSIM), as well as in subjective evaluations by showing better visual quality. Moreover, the editable and self-labeled image generation capabilities make it a valuable tool in addressing challenges like occlusion, deformation, and varying environmental conditions in the 2D object detection.