Preserving hand-drawn qualities in audiovisual performance through sketch-based interaction

Abstract

Live coding – the real-time procedural creation of audiovisual works – suggests opportunities to extend hand-drawn animation; however, existing live coding systems are incompatible with manual animation workflows. Manual input is not a primary datatype in existing live coding languages and live coding tools require using symbolic programming environments. We theorize that by applying direct manipulation to the domain of live coding, we can enable animators to create expressive mappings between hand-drawn animations and audio effects in realtime. We present Megafauna, a sketch-based system for audiovisual performance, informed by interviews with professional animators. Megafauna supports the integrated generation and control of hand-drawn animation and audio sequences by enabling animators to directly sketch mapping functions between animation frames and sound generators. We demonstrate the expressive potential of Megafauna by reproducing animated compositions from procedural and manual domains. We evaluate the opportunities of our approach for live production through an expert review of a performance piece created with Megafauna.