Flexible content-aware image synthesis for maritime tasks with diffusion models

Abstract

Maritime environmental perception suffers greatly from data lack due to the high cost of data collection at sea. In this paper, a novel image synthesis method is proposed to automatically generate target images with diverse foreground and background. Specifically, foreground images for various poses are generated using a diffusion model, presenting different modalities of the detected target. The environment conditions of the background images are flexibly adjusted by inputting semantic prompts to another diffusion model. Then a 3D affine diffusion model is proposed for effective fusion of foreground and background. This module calculates the size and position of the foreground image within the background image through affine transformation, and utilizes the excellent image fusion ability of the diffusion model to achieve high-quality image synthesis. As a result, a set of dynamically variable foreground and background images are generated to increase the pose and weather diversity of maritime object detection samples. Extensive experiments are conducted to verify the effectiveness of image synthesis algorithms, and this method can also serve downstream tasks, effectively improving the accuracy of maritime environmental perception algorithms. The code is available at https://github.com/xuezhen2018/flexible_content_aware_image_synthesis.