Attaching the Value of Sensorial Experience to Pervasive Multimedia Applications

We use sensory data to improve and make multimedia applications pervasive. The learning cycle is used as a centrepiece from which the user experience is derived and higher-level knowledge understood. We propose an architecture by which this sensorial experience can be presented to users, creating immersion and continuity. I. BUILDING UP THE EXPERIENCE he ability to understand and also to empathize or interact with a user requires access to the deeper levels of the user's emotions and expressions. By understanding people’s feelings, dreams and thoughts, we can create a truly interactive conversation with them. However, an experience is never static; on the contrary, it evolves. To gather knowledge on these subjects we need to record data from sensors at the place and time of their occurrence. Sanders [1] and Cain [2] suggest learning from people – summarized by figure 1. Listening to what people say and think provides us with an explicit knowledge (beyond what they express in words). Watching what people do and seeing what they use provides us with observable knowledge. Understanding what people think, know and dream provides us with tacit knowledge (knowledge that can’t be expressed in words). This information can show us how a person envisions a better future through expressed latent needs (only satisfiable in the future). Figure 1. Three levels of knowledge extraction from observing human actions Kolb [3] associates human experience with the learning process. He suggests there are four stages following one another (figure 2): the Concrete Experience; followed by Reflection on that experience; with Abstract Conceptualization then deriving social judgment to describe the experience; leading to ways of modifying the next experience called Active Experimentation, that feeds the next Concrete Experience. This recursive experiencing may happen very fast or over longer periods, depending on the case. II.FROM SENSING TO EXPERIENCING A. The Notion of Sensing Aristotle defined the traditional five human senses as sight, hearing, touch, smell and taste [4]. Humans have six additional senses, namely pain (nociception), balance (equilibrioception), joint motion and acceleration (proprioception & kinesthesia), sense of time, sense of temperature differences (thermoception), and the sense of direction (magnetoception) [5]. All those senses actively participate in the creation of the user experience. Therefore, when designing application for the end-user, it is crucial to record the sensed aspects of the experience, with samples of the before, during and after, regardless of the physical setting of the observed experience. B. Raw Sensing and Mobile Devices To record the physical aspects of a user experience – such as ambient sounds, people presence and location – physical sensors are necessary. To achieve this regardless of the physical setting, a pervasive approach must be taken. Mobile devices (such as cellular phones) have become an important piece in ubiquitous experiences, offering services to people whenever and wherever they are. Moreover, mobile devices are becoming a convergence point for many different sensors, ranging from light, motion, touch, location and proximity. These are typically included in shipped products to satisfy the needs of a specific vertical application. However, the growing number of specific sensor-oriented applications in a mobile device, and improved developer access to Application Programming Interfaces (API), enable this sensory information to be combined to provide additional information for novel applications and services. The MyLifeBits project [6] exemplifies research attempts to exploit these developments for a form of life logging [7]. Sensors are also being embedded into many physical environments to form the so-called Internet of Things. This trend is primarily driven by industrial process automation and academia with increasing numbers of consumer electronics applications [8]. It is also gaining momentum T Attaching the Value of Sensorial Experience to Pervasive Multimedia Applications Adrian Hornsby , Rod Walsh 2 1 Tampere University of Technology, Finland 2 Nokia Research Center, Tampere, Finland Figure 2. A simplified leaning cycle 978-1-4244-5176-0/10/$26.00 ©2010 IEEE This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE CCNC 2010 proceedings