StyleTune: Interactive Style Transfer Enhancement on Mobile Devices

Abstract

We present StyleTune, a mobile app for interactive style transfer enhancement that enables global and spatial control over stroke elements and can generate high fidelity outputs. The app uses adjustable neural style transfer (NST) networks to enable art-direction of stroke size and orientation in the output image. The implemented approach enables continuous and seamless edits through a unified stroke-size representation in the feature space of the style transfer network. StyleTune introduces a three-stage user interface, that enables users to first explore global stroke parametrizations for a chosen NST. They can then interactively locally retouch the stroke size and orientation using brush metaphors. Finally, high resolution outputs of 20 Megapixels and more can be obtained using a patch-based upsampling and local detail transfer approach, that transfers small-scale details such as paint-bristles and canvas structure. The app uses Apple’s CoreML and Metal APIs for efficient on-device processing.