Using Abstract Anchors for Automatic Authoring of Sensory Effects Based on Ambient Sound Recognition

Abstract

Mulsemedia applications are shown to improve user experience by synchronizing sensory effects along with audiovisual content. An usual approach for specifying such synchronization is to mark the moments when a given effect has to be executed in the main media object. Multimedia authoring languages provide a way to mark moments of a media object through anchors. An anchor indicates a temporal portion (video frames or audio samples) in relation to its parent media object. However, to synchronize such portions with sensory content can be costly. When, for example, a scene component appear multiple times on the media content. This problem was tackled in previous work by providing an approach for creating abstract anchors in declarative multimedia documents. An abstract anchor represents (possibly) several media anchors. By using a scene recognition software, the anchors where duplicated whenever the same component was identified on the main media. Such approach, was proven useful for creating automatic sensory effects based on video recognition. This paper extends the proposal of abstract anchors in order to enable the automatic generation of sensory effects using sound recognition. This paper also extends the Abstract Anchor Processor to implement media pre-processing and to adapt the sound recognition software response.