{"title":"Dreamsound: Deep Activation Layer Sonification","authors":"Federico Nicolás Cámara Halac, Matías Delgadino","doi":"10.21785/icad2021.032","DOIUrl":"https://doi.org/10.21785/icad2021.032","url":null,"abstract":"Deep learning (DL) in audio signal processing has received much attention in the last four years, and it is still a growing field. Some attempts were made to translate deep networks from image to audio applications, and we discuss some of the issues that arise. In the present paper, we present DreamSound, a creative adaptation of Deep Dream to sound addressed from two approaches: input manipulation, and sonification design. Our chosen model is YAMNet, a pre-trained deep network for sound classification. We present the original Deep Dream activation maximization function in relation to three filter-based creative variations that were implemented. We find that more interesting results are achieved with this filter-based approach, but that there is still room for experimentation.","PeriodicalId":448753,"journal":{"name":"Proceedings of the 26th International Conference on Auditory Display (ICAD 2021)","volume":"31 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128950971","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Layers of Meaning: The Ocean’s Natural Acoustics and the Music of Its Datasets","authors":"Heather R. Spence, M. Ballora","doi":"10.21785/icad2021.017","DOIUrl":"https://doi.org/10.21785/icad2021.017","url":null,"abstract":"The transdisciplinary National Academies Keck Futures Initiative (NAKFI) conference on the Deep Blue Sea sparked a collaboration between sonification expert Mark Ballora and marine biologist and sound artist Heather Spence. Research involving long-term Marine Passive Acoustic Monitoring (MPAM) of the MesoAmerican Reef system forms the basis for a gradient of audio products: 1) layering a tour guide acoustic instrument over raw and manipulated soundscape recordings; 2) layering of multiple acoustic instruments over duty cycle interpretation sampling; and 3) layering of data sonification over the original data, with additional acoustic instrument layers. The audio products are designed to promote data exploration and understanding by researchers and students, as well as an emotional impact musically with conservation themes. Presentations have included live and virtual performances and workshops. Next steps include sonification of other correlated environmental data with the original sound data in raw, manipulated, and sonified forms.","PeriodicalId":448753,"journal":{"name":"Proceedings of the 26th International Conference on Auditory Display (ICAD 2021)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129872043","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"The Design of a Smart City Sonification System Using a Conceptual Blending and Musical Framework, Web Audio and Deep Learning Techniques","authors":"S. Roddy, B. Bridges","doi":"10.21785/icad2021.010","DOIUrl":"https://doi.org/10.21785/icad2021.010","url":null,"abstract":"This paper describes an auditory display system for smart city data for Dublin City, Ireland. It introduces and describes the different layers of the system and outlines how they operate individually and interact with one another. The system uses a deep learning model called a variational autoencoder to generate musical content to represent data points. Further data-to-sound mappings are introduced via parameter mapping sonification techniques during sound synthesis and post-processing. Conceptual blending and music theory provide frameworks, which govern the design of the system. The paper ends with a discussion of the design process that contextualizes the contribution, highlighting the interdisciplinary nature of the project, which spans data analytics, music composition and human-computer interaction.","PeriodicalId":448753,"journal":{"name":"Proceedings of the 26th International Conference on Auditory Display (ICAD 2021)","volume":"36 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132695524","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Strategies and Tools for the Sonification of Prosodic Data: A Composer's Perspective","authors":"Fabio C. Ciardi","doi":"10.21785/icad2021.041","DOIUrl":"https://doi.org/10.21785/icad2021.041","url":null,"abstract":"Does it make sense to sonify information that refers to an already audible phenomenon such as prosodic data? In order to be useful, a sonification of prosody should contribute to the comprehension of paralinguistic features that may not otherwise attract the attention of the listener. Within this context, this paper illustrates a modular and flexible framework for the reduction and processing of prosodic data to be used for enhancing the perception of a speaker's intention, attitude and emotions. The model uses speech audio as input and provides MIDI and MusicXML data as output so allowing samplers and notation software to auralize and display the information. The described architecture has been subjectively tested by the author over many years in compositions for solo instruments, ensembles and orchestra. Two outcomes of the research are discussed: the advantages of an adaptive strategy for data reduction, and the auditory display of the deep pitch and temporal structures underlying prosodic processing.","PeriodicalId":448753,"journal":{"name":"Proceedings of the 26th International Conference on Auditory Display (ICAD 2021)","volume":"4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126118983","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Konstantinos Bakogiannis, S. Polychronopoulos, Areti Andreopoulou, A. Georgaki
{"title":"Data and Information Transmission in the Context of Sonfication","authors":"Konstantinos Bakogiannis, S. Polychronopoulos, Areti Andreopoulou, A. Georgaki","doi":"10.21785/icad2021.022","DOIUrl":"https://doi.org/10.21785/icad2021.022","url":null,"abstract":"This paper illustrates the significance of the concept of information as a tool to expound sonification design. Previous works approached the concept of information systematically. However, its structural characteristics during the process of sonification have not been thoroughly discussed. In order to address the above, this paper presents a framework based on the definition of information from the fields of physics, communication engineering, cybernetics, and systems theory. According to this framework, the representation of a phenomenon into organized sound becomes possible by the propagation of organization within the components of the sonification communication model. Moreover, a distinction between the terms data and information is proposed and related to well-established sonification techniques (Audification, PMSon, MBS). The structural characteristics of the phenomenon (described in terms of entropy) are linked with sonification functions leading to new perspectives of sonification design.","PeriodicalId":448753,"journal":{"name":"Proceedings of the 26th International Conference on Auditory Display (ICAD 2021)","volume":"68 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131367147","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"ATHsENSe: A Multisensory Outdoor Installation","authors":"Alexandros Drymonitis, D. Charitos","doi":"10.21785/icad2021.035","DOIUrl":"https://doi.org/10.21785/icad2021.035","url":null,"abstract":"The ATHsENSe installation is a four part AV installation based on environmental data from the city of Athens. It comprises four different installations: a visualization projection of the data, a sound installation where the data are being sonified, an installation comprising eight LED displays projecting texts from users of the web app of ATHsENSe, and a light installation reflecting the sound levels of a central point of Athens. The second of the four installations is presented here where data concerning CO, CO2, humidity and others, together with texts provided by the web app users, are being sonified in a six-channel surround installation. In addition to the electronic sounds of the sonification, four kinetic sculptures are being triggered by the system which create acoustic sounds by hitting found objects. This installation is located at the Serafeio complex in Athens, and was commissioned to the Spatial Research Media Group by the municipality of Athens after their successful participation in the Interventions in the City competition.","PeriodicalId":448753,"journal":{"name":"Proceedings of the 26th International Conference on Auditory Display (ICAD 2021)","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122271675","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Mariana Seiça, Licinio Gomes Roque, Pedro Martins, F. A. Cardoso
{"title":"A Systemic Perspective for Sonification Aesthetics","authors":"Mariana Seiça, Licinio Gomes Roque, Pedro Martins, F. A. Cardoso","doi":"10.21785/icad2021.033","DOIUrl":"https://doi.org/10.21785/icad2021.033","url":null,"abstract":"For more than twenty-five years, the sonification field has been attempting to establish itself as a primary body of knowledge communicating through sound. Despite multiple efforts to embrace the interdisciplinary nature of the field and the subjective nature of sound, we wonder: is the tendency for dealing with such challenges through an objective, functional communication, with a single interpretation criterion, limiting the epistemic boundaries of action? How can a subjectively perceived medium such as sound be embraced in all its aesthetic dimensions? We propose a conceptual transition through the reframing of a sonification as a living system for creating aesthetic experiences. This will be achieved by drawing notions from phenomenology, embodied perception, human-computer interaction and soundscape theory. A systemic sonification distinguishes itself as an ever-evolving system built on dynamic structures that actively responds to changes in its environment and interactions from surrounding beings. Driven by a series of emerging concepts of non-linearity, networks, nested systems and intertwined relationships, the system’s resilience and adaptability grows with each interaction, recentring the human protagonist as the weaver of his/her aesthetic experience through a self-transcendent process that expands the perception field.","PeriodicalId":448753,"journal":{"name":"Proceedings of the 26th International Conference on Auditory Display (ICAD 2021)","volume":"279 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116436692","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"High-Order Surrogacy for the Audiovisual Display of Dance","authors":"Stefano Kalonaris, I. Zannos","doi":"10.21785/icad2021.001","DOIUrl":"https://doi.org/10.21785/icad2021.001","url":null,"abstract":"The current pandemic (COVID-19) has had considerable impact on many fronts, not least on the physical presence of humans, affecting how we relate to one another and to the natural environment. To investigate these two interactions, the notion of surrogacy, originally described by Smalley as remoteness between source and sonic gesture, is considered and extended to include bodily gesture, for the rendering of contemporary dance performances into abstract audiovisual compositions/objects. To this end, for a given dance performance, sonification of the motion capture data is combined with video-frame processing of the video recording. In this study, we focus on higher order surrogacy and associate this with 1) a soundscape ecology-inspired approach to sonification, whereby three species of sounds coexist and adapt in the environment according to the symbiotic paradigm of mutualism, and 2) a wave space method to sonify their coevolution. Aesthetic implications of this procedure in the context of multimodal, telematic/remote and virtual systems are discussed as disembodied presence emerges as a dominant trope in our daily experience.","PeriodicalId":448753,"journal":{"name":"Proceedings of the 26th International Conference on Auditory Display (ICAD 2021)","volume":"5 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123728809","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Sonification for Everyone Everywhere - Evaluating the WebAudioXML Sonification Toolkit for Browsers","authors":"Hans Lindetorp, Kjetil Falkenberg","doi":"10.21785/icad2021.009","DOIUrl":"https://doi.org/10.21785/icad2021.009","url":null,"abstract":"Creating an effective sonification is a challenging task that requires skills and knowledge on an expertise level in several disciplines. This study contributes with WebAudioXML Sonification Toolkit (WAST) that aims at reaching new groups who have not yet considered themselves to be part of the ICAD community. We have designed, built, and evaluated the toolkit by analysing ten student projects using it and conclude that WAST did meet our expectations and that it lead to students taking a deep approach to learning and successfully contributed to reaching the learning outcomes. The result indicates that WAST is both easy-to-use, highly accessible, extensively flexible and offers possibilities to share the sonification in any device’s web browser simply through a web link, and without installations. We also suggest that a sonification toolkit would become an even more creative environment with virtual instruments and mixing features typically found in Digital Audio Workstations.","PeriodicalId":448753,"journal":{"name":"Proceedings of the 26th International Conference on Auditory Display (ICAD 2021)","volume":"49 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122862301","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Lucas Temor, D E Macdonald, T. Natarajan, P. Coppin
{"title":"Perceptually-Motivated Sonification of Spatiotemporally-Dynamic CFD Data","authors":"Lucas Temor, D E Macdonald, T. Natarajan, P. Coppin","doi":"10.21785/icad2021.039","DOIUrl":"https://doi.org/10.21785/icad2021.039","url":null,"abstract":"Everyday perception and action are fundamentally multisensory. Despite this, the sole reliance on visualization for the representation of complex 3D spatiotemporal data is still widespread. In the past we have proposed various prototypes for the sonification of dense data from computational fluid dynamics (CFD) simulations of turbulent-like blood flow, but did not robustly consider the perception and associated meaning-making of the resultant sounds. To reduce some of the complexities of these data for sonification, in this work we present a feature-based approach, applying ideas from auditory scene analysis to sonify different data features along perceptually-separable auditory streams. As there are many possible features in these dense data, we followed the analogy of “caricature” to guide our definition and subsequent amplification of unique spectral and fluctuating features, while effectively minimizing the features common between simulations. This approach may allow for better insight into the behavior of flow instabilities when compared to our previous sonifications and/or visualizations, and additionally we observed benefits when some redundancy was maintained between modalities.","PeriodicalId":448753,"journal":{"name":"Proceedings of the 26th International Conference on Auditory Display (ICAD 2021)","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124247066","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}