Jukepix: A Cross-Modality Approach to Transform Paintings into Music Segments

Abstract

The challenges in transforming paintings into music is well-known, since the relationship between two kinds of art is unclear. Different composers write different music when the same painting is presented to them. In this paper, a cross-mordality model has been proposed for transforming images into multitrack music based on the framework of deep convolutional generative adversarial networks (DCGANs). The proposed model is trained on a classical music dataset and a dataset of impressionist paintings. The model can be applied to transfer impressionist paintings into classical music with two tracks. By using music evaluation methods, the harmonicity of the generated music can be confirmed. Our model is the first attempt of our knowledge at transforming paintings into music segments.