iBaton: conducting virtual concerts using smartphones

Abstract

With the emergence of virtual environments, even ordinary computer users can drag-and-drop 3d objects (cities, buildings, furniture, instruments, controls, animals and avatars) from galleries and arrange them to create attractive cyber-worlds. We have created a virtual concert application using Alice, a 3d rapid prototyping programming environment, in which instruments are arranged around a virtual conductor (in this case the user) located at their center. A user-conductor with a smartphone can use it as a simplified baton, pointing at a preferred instrument and tapping to select or start playing. When selected, an instrument is jiggled or its components dilated and contracted, and a spotlight appears until the instrument is muted, providing the conductor and audience with visual cues about the ensemble. One can also adjust volume/panning of a selected instrument by sliding corresponding controls on the smart-phone screen. Alice 3 (Beta) provides a rich api to control and display objects, but native audio capabilities are limited to playing an audio file and adjusting individual volume before playing. Using a plugin for NetBeans, Alice programs (scenarios) can be edited as Java code outside Alice. With the Java Sound api, which provides more complete control for controlling sampled audio and midi, we implement audio-related functions to the virtual concert. Smartphones have magnetometers that can be used to detect yaw, which data is acquired through a compatible programming language and library. Spatial information of smartphones can be accessed using the platform's api and transmitted to a collaborative virtual environment (cve) session server via Wi--Fi. A middleware program, the "Alice--Cve Bridge," retrieves these events through the server and selects/plays/adjusts instruments in the Alice-rendered virtual environment. Such explorations suggest the power of emerging mobile devices as generalized remote controls for interactive multimedia and ubiquitous computing.