The SIT book: audio as affective imagery for interactive storybooks

Abstract

We describe a working prototype built as part of our continuing research focus on new document genres and the crossmodal affordances of interactive audio. Our experimental SIT (Sound-Image-Text) Book is a personal interactive reading experience that combines the look and feel of a real book -- a beautiful cover, paper pages and printed images and text -- with the rich, evocative quality of a movie soundtrack. The soundtrack is multi-track and includes music and sound effects. The SIT Book uses electric field sensors located in the book binding to sense the reader's casual book handling and fingering of the page; these sensors control the ambient audio. The particular point of the SIT Book is to explore the use of background sound to provide a sense of place, and to add affectto the experience of reading a book without interrupting the flow of the story.