The Handbook of Multimodal-Multisensor Interfaces, Volume 1最新文献

{"title":"Gaze-informed multimodal interaction","authors":"Pernilla Qvarfordt","doi":"10.1145/3015783.3015794","DOIUrl":"https://doi.org/10.1145/3015783.3015794","url":null,"abstract":"Observe a person pointing out and describing something. Where is that person looking? Chances are good that this person also looks at what she is talking about and pointing at. Gaze is naturally coordinated with our speech and hand movements. By utilizing this tendency, we can create a natural interaction with computing devices and environments. In multimodal gaze interaction, data from eye trackers are used as an active input mode, where, for instance, gaze is used as an alternative, or complementary, pointing modality along with other input modalities. Using gaze as an active, or explicit, input method is challenging for several reasons. One of them being that eyes are primarily used for perceiving our environment, so knowing when a person selects an item with gaze versus just looking around is an issue. Researchers have tried to solve this by combining gaze with various input methods, such as manual pointing, speech, touch, etc.However, gaze information can also be used in interactive systems for other purposes than explicit pointing, since a user's gaze is a good indication of the user's attention. In passive gaze interaction, the gaze is not used as the primary input method, but as a supporting one. In these kinds of systems, gaze is mainly used for inferring and reasoning about the user's cognitive state or activities in a way that can support the interaction. These kinds of multimodal systems often combine gaze with a multitude of input modalities. One example is to detect what features in an image a person is looking for, and use this information to suggest regions or other images that the person has not yet seen.In this chapter, eye movement and eye tracking data analysis is first reviewed (Section 9.2), followed by a discussion of eye movements in relation to other modalities (Section 9.3) to gain basic knowledge about eye tracking and gaze behavior. In Section 9.4, systems using gaze as active or passive input methods are discussed. Finally, Section 9.5 concludes the chapter.","PeriodicalId":222911,"journal":{"name":"The Handbook of Multimodal-Multisensor Interfaces, Volume 1","volume":"81 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-04-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114479212","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A background perspective on touch as a multimodal (and multisensor) construct","authors":"K. Hinckley","doi":"10.1145/3015783.3015789","DOIUrl":"https://doi.org/10.1145/3015783.3015789","url":null,"abstract":"This chapter will illustrate, through a series of examples, seven different perspectives of how touch input can be re-framed and re-conceived as a multimodal, multisensor construct. \u0000 \u0000These perspectives often can particularly benefit from considering the background of interaction [Buxton 1995]---that is, interaction that takes place \"behind\" the foreground of the user's conscious attention, in response to sensed contextual information. For example, with touch-screen input, the user's intentional contact with the screen would comprise the foreground act, but the resulting vibrational forces imparted to the device can be sensed and leveraged \"in the background\" to infer additional contextual details of the touch. \u0000 \u0000Over the years, I've found this background perspective extremely useful as a tool-for-thought to devise novel interactions, especially when multiple modalities and multiple sensors can be used simultaneously, in complementary and mutually reenforcing ways. This approach can be especially helpful to break out of whatever preconceptions one might have regarding an input modality, even something as 144 Chapter 4 A Background Perspective on Touch as a Multimodal (and Multisensor) Construct seemingly well studied and well understood as touch, which just for that reason will provide us with the bulk of the examples that I draw from in this chapter. \u0000 \u0000These perspectives of touch range from its traditional view as a modality that affords direct and intentional touchscreen input, to the inadvertent (yet still potentially valuable) phenomenon of unintentional (or \"Midas\") touch. We will consider various combinations of touch with other sensor signals such as tilt, inertial motion, grip sensing, and above-screen pre-touch sensing. We'll also discuss interesting ways to use pen and touch as complementary modalities for bimanual interaction. The focus of the chapter is design-centric, with contributions that focus on invention and innovation [Hudson and Mankoff 2014], rather than contributions, say, of formal experimental analysis or recognition methodologies. Likewise, the focus here is on the input side of touch, rather than the output side---as afforded by haptic and tactile feedback technologies (cf. Chapter 3). \u0000 \u0000The key point of this chapter is that low-level \"sensing\" channels such as grip and proximity and motion can be conceived of as new modalities that afford natural interaction with devices. And, likewise, it is possible to subtly shift our perspective of \"modalities\" to consider novel ways that they may yield insights of sensing, particularly if considered from the oft-neglected perspective that the background of interaction can support the user's task focus in the foreground of attention most effectively. \u0000 \u0000To encourage reflection on the material, the chapter concludes with eight openended Focus Questions that can also serve as starting-points for independent research projects, as well as a Glossary of key terms.","PeriodicalId":222911,"journal":{"name":"The Handbook of Multimodal-Multisensor Interfaces, Volume 1","volume":"16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-04-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124577161","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The impact of multimodal-multisensory learning on human performance and brain activation patterns","authors":"K. James, S. Vinci-Booher, Felipe Munoz-Rubke","doi":"10.1145/3015783.3015787","DOIUrl":"https://doi.org/10.1145/3015783.3015787","url":null,"abstract":"The human brain is inherently a multimodal-multisensory dynamic learning system. All information that is processed by the brain must first be encoded through sensory systems and this sensory input can only be attained through motor movement. Although each sensory modality processes different signals from the environment in qualitatively different ways (e.g., sound waves, light waves, pressure, etc.), these signals are transduced into a common language in the brain. The signals are then associated and combined to produce our phenomenology of a coherent world. Therefore, the brain processes a seemingly unlimited amount of multisensory information for the purpose of interacting with the world. This interaction with the world, through the body, is multimodal. The body allows one to affect the environment through multiple motor movements (hand movements, locomotion, speech, gestures, etc.). These various actions, in turn, shape the multisensory input that the brain will subsequently receive. The feedforward-feedback loop that occurs every millisecond among sensory and motor systems is a reflection of these multisensory and multimodal interactions among the brain, body, and environment. As an aid to comprehension, readers are referred to this chapter's Focus Questions and to the Glossary for a definition of terminology. \u0000 \u0000In the following, we begin by delving deeper into how sensory signals are transduced in the brain and how multimodal activity shapes signal processing. We then provide samples of research that have demonstrated that multimodal interactions with the world, through action, facilitate learning. An overview of research on performance measured by overt behavioral responses in adult and developing populations is followed by examples of research on the effects that multimodal learning has on brain plasticity in adults and children. Together, the behavioral and neuroimaging literature underscore the importance of learning through multimodalmultisensory interactions throughout human development.","PeriodicalId":222911,"journal":{"name":"The Handbook of Multimodal-Multisensor Interfaces, Volume 1","volume":"27 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-04-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121422534","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Multisensory haptic interactions: understanding the sense and designing for it","authors":"Karon E Maclean, Oliver S. Schneider, H. Seifi","doi":"10.1145/3015783.3015788","DOIUrl":"https://doi.org/10.1145/3015783.3015788","url":null,"abstract":"Our haptic sense comprises both taction or cutaneous information obtained through receptors in the skin, and kinesthetic awareness of body forces and motions. Broadly speaking, haptic interfaces to computing systems are anything a user touches or is touched by, to control, experience, or receive information from something with a computer in it. Keyboard and mouse, a physical button on a kitchen blender, and the glass touchscreen on your smartphone are energetically passive haptic interfaces: no external energy is pumped into the users' body from a powered actuator. Most readers will have encountered energetically active haptic feedback as a vibrotactile (VT) buzz or forces in a gaming joystick, a force feedback device in a research lab, or a physically interactive robot. Much more is possible. \u0000 \u0000When we bring touch into an interaction, we invoke characteristics that are unique or accentuated relative to other modalities. Like most powerful design resources, these traits also impose constraints. The job of a haptic designer is to understand these \"superpowers\" and their costs and limits, and then to deploy them for an optimally enriched experience. \u0000 \u0000Both jobs are relatively uncharted, even though engineers have been building devices with the explicit intention of haptic display for over 25 years, and psychophysicists have been studying this rich, complex sense for as many decades. What makes it so difficult? Our haptic sense is reallymanydifferent senses, neurally integrated; meanwhile, the technology of haptic display is anything but stable, with engineering challenges of a different nature than those for graphics and sound. In the last few years, technological advances from materials to robotics have opened new possibilities for the use of energetically active haptics in user interfaces, our primary focus here. Needs are exposed at a large scale by newly ubiquitous technology like \"touch\" screens crying out for physical feedback, and high-fidelity virtual reality visuals that are stalled in effectiveness without force display.","PeriodicalId":222911,"journal":{"name":"The Handbook of Multimodal-Multisensor Interfaces, Volume 1","volume":"60 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-04-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126457381","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Audio and visual modality combination in speech processing applications","authors":"G. Potamianos, E. Marcheret, Youssef Mroueh, V. Goel, Alexandros Koumbaroulis, Argyrios S. Vartholomaios, Spyridon Thermos","doi":"10.1145/3015783.3015797","DOIUrl":"https://doi.org/10.1145/3015783.3015797","url":null,"abstract":"Chances are that most of us have experienced difficulty in listening to our interlocutor during face-to-face conversation while in highly noisy environments, such as next to heavy traffic or over the background of high-intensity speech babble or loud music. In such occasions, we may have found ourselves looking at the speaker's lower face, while our interlocutor articulates speech, in order to help us enhance speech intelligibility. In fact, what we resort to in such circumstances is known as lipreading or speechreading, namely the recognition of the so-called \"visual speech modality\" and its combination (fusion) with the available noisy audio data. \u0000 \u0000Similar to humans, automatic speech recognition (ASR) systems also face difficulties in noisy environments. In recent years, ASR technology has made remarkable strides following the adoption of deep-learning techniques [Hinton et al. 2012, Yu and Deng 2015]. This has led to advanced ASR systems bridging the gap with human performance [Xiong et al. 2017], compared to their significant lag 20 years earlier, as established by Lippmann [1997]. Nevertheless, the quest for ASR noise robustness, particularly when noise is non-stationary and mismatched to training data, remains an active research topic [Li et al. 2015]. \u0000 \u0000To help us mitigate the aforementioned problem, the question naturally arises as to whether or not machines can be designed to mimic human speech perception in noise. Namely, can they successfully incorporate visual speech into the ASR pipeline, especially since this represents an additional information source unaffected by the acoustic environment. At Bell-Labs, Petajan [1984] was the first to develop and implement an early audio-visual automatic speech recognition (AVASR) system. Since then, the area has witnessed significant research activity, paralleling the advances in traditional audio-only ASR, while also utilizing progress in the computer vision and machine learning fields. Not surprisingly, adoption of deep learning techniques has created renewed interest in the field, resulting in remarkable progress on challenging domains, even surpassing human lipreading performance [Chung et al. 2017]. \u0000 \u0000Since the very early works in the field [Stork and Hennecke 1996], design of AVASR systems has generally followed the basic architecture of Figure 12.1. There, a visual front-end module is depicted to provide speech-informative features that are extracted from the video of the speaker's face. These are subsequently fused with acoustic features into the speech recognition process. Clearly, compared to audio-only ASR, visual speech information extraction and audio-visual fusion (or integration) constitute two additional distinct components on which to focus. Indeed, their robustness under a wide range of audio-visual conditions and their efficient implementation represent significant challenges that, to date, remain the focus of active research. It should be noted that rapid recent advances, ","PeriodicalId":222911,"journal":{"name":"The Handbook of Multimodal-Multisensor Interfaces, Volume 1","volume":"161 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-04-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134500096","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Multimodal gesture recognition","authors":"Athanasios Katsamanis, Vassilis Pitsikalis, Stavros Theodorakis, P. Maragos","doi":"10.1145/3015783.3015796","DOIUrl":"https://doi.org/10.1145/3015783.3015796","url":null,"abstract":"Starting from the famous \"Put That There!\" demonstration prototype, developed by the Architecture Machine Group at MIT in the late 1970s, the growing potential of multimodal gesture interfaces in natural human-machine communication setups has stimulated people's imagination and motivated significant research efforts in the fields of computer vision, speech recognition, multimodal sensing, fusion, and human-computer interaction (HCI). In the words of Bolt [1980, p. 1]: \"Because voice can be augmented with simultaneous pointing, the free usage of pronouns becomes possible, with a corresponding gain in naturalness and economy of expression. Conversely, gesture aided by voice gains precision in its power to reference.\" \u0000 \u0000Multimodal gesture recognition lies at the heart of such interfaces. As also defined in the Glossary, the term refers to the complex computational task comprising three main modules: (a) tracking of human movements, primarily of the hands and arms, and recognition of characteristic such motion patterns; (b) detection of accompanying speech activity and recognition of what is spoken; and (c) combination of the available audio-visual information streams to identify the multimodally communicated message. \u0000 \u0000To successfully perform such tasks, the original \"Put That There!\" system of Bolt [1980] imposed certain limitations on the interaction. Specifically, it required that the user be tethered by wearing a position sensing device on the wrist to capture gesturing and a headset microphone to record speech, and it allowed multimodal manipulation via speech and gestures of a small only set of shapes on a rather large screen (see also Figure 11.1). Since then, however, research efforts in the field of multimodal gesture recognition have moved beyond such limited scenarios, capturing and processing the multimodal data streams by employing distant audio and visual sensors that are unobtrusive to humans. In particular, in recent years, the introduction of affordable and compact multimodal sensors like the Microsoft Kinect has enabled robust capturing of human activity. This is due to the wealth of raw and metadata streams provided by the device, in addition to the traditional planar RGB video, such as depth scene information, multiple audio channels, and human skeleton and facial tracking, among others [Kinect 2016]. Such advancements have led to intensified efforts to integrate multimodal gesture interfaces in real-life applications. \u0000 \u0000Indeed, the field of multimodal gesture recognition has been attracting increasing interest, being driven by novel HCI paradigms on a continuously expanding range of devices equipped with multimodal sensors and ever-increasing computational power, for example smartphones and smart television sets. Nevertheless, the capabilities of modern multimodal gesture systems remain limited. In particular, the set of gestures accounted for in typical setups is mostly constrained to pointing gestures, a number of emblemati","PeriodicalId":222911,"journal":{"name":"The Handbook of Multimodal-Multisensor Interfaces, Volume 1","volume":"32 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-04-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124912452","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Understanding and supporting modality choices","authors":"A. Jameson, P. Kristensson","doi":"10.1145/3015783.3015790","DOIUrl":"https://doi.org/10.1145/3015783.3015790","url":null,"abstract":"One of the characteristic benefits of multimodal-multisensor processing is that it gives users more freedom of choice than they would otherwise have. The most central type of choice concerns the use of input modalities: When performing a particular task with a multimodal system, a user will often have a set of two or more input modalities to choose from. She may choose to employ just one of these; or she may use two or more modalities simultaneously or in alternation. In general, we can say that she must somehow choose some combination of modalities from the available set. \u0000 \u0000It has often been pointed out in the multimodal literature (e.g., by Oviatt and Cohen [2015, chap. 3]) that this freedom of choice is a major benefit of multimodality in that it gives users the flexibility to deal with a wide range of needs and situations. But this additional freedom of choice can be beneficial to users only if they are able to make reasonably good modality choices. If they make choices that actually lead to less satisfactory results than they could get with a system that offered more limited interaction possibilities, then multimodality will be a drawback rather than an advantage. \u0000 \u0000We begin this chapter by summarizing what can be learned from the research literature about the modality choices that people make, concluding by arguing that there is a need to focus more on the processes by which people arrive at such choices and on strategies for helping them to choose more effectively. In Sections 5.3--5.8, we introduce relevant psychology-based models for this purpose and illustrate their applicability to the problem of modality choice.","PeriodicalId":222911,"journal":{"name":"The Handbook of Multimodal-Multisensor Interfaces, Volume 1","volume":"137 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-04-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133098562","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Theoretical foundations of multimodal interfaces and systems","authors":"S. Oviatt","doi":"10.1145/3015783.3015786","DOIUrl":"https://doi.org/10.1145/3015783.3015786","url":null,"abstract":"This chapter discusses the theoretical foundations of multisensory perception and multimodal communication. It provides a basis for understanding the performance advantages of multimodal interfaces, as well as how to design them to reap these advantages. Historically, the major theories that have influenced contemporary views of multimodal interaction and interface design include Gestalt theory, Working Memory theory, and Activity theory. They include perception-action dynamic theories and also limited resource theories that focus on constraints involving attention and short-term memory. This chapter emphasizes these theories in part because they are supported heavily by neuroscience findings. Their predictions also have been corroborated by studies on multimodal human-computer interaction. In addition to summarizing these three main theories and their impact, several related theoretical frameworks will be described that have influenced multimodal interface design, including Multiple Resource theory, Cognitive Load theory, Embodied Cognition, Communication Accommodation theory, and Affordance theory. \u0000 \u0000The large and multidisciplinary body of research on multisensory perception, production, and multimodal interaction confirms many Gestalt,Working Memory, and Activity theory predictions that will be discussed in this chapter. These theories provide conceptual anchors. They create a path for understanding how to design more powerful systems, so we can gain better control over our own future. In spite of this, it is surprising how many systems are developed from an engineering perspective that is sophisticated, yet in a complete theoretical vacuum that Leonardo da Vinci would have ridiculed: \u0000 \u0000Those who fall in love with practice without science are like a sailor who enters a ship without helm or compass, and who never can be certain whither he is going. Richter and Wells [2008] \u0000 \u0000This chapter aims to provide a better basis for motivating and accelerating future multimodal system design, and the quality of its impact on human users. [email protected] a definition of highlighted terms in this chapter, see the Glossary. For other related terms and concepts, also see the textbook on multimodal interfaces by [Oviatt and Cohen 2015]. Focus Questions to aid comprehension are available at the end of this chapter.","PeriodicalId":222911,"journal":{"name":"The Handbook of Multimodal-Multisensor Interfaces, Volume 1","volume":"108 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-04-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122535817","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Multimodal feedback in HCI: haptics, non-speech audio, and their applications","authors":"Euan Freeman, Graham A. Wilson, Dong-Bach Vo, Alexander Ng, I. Politis, S. Brewster","doi":"10.1145/3015783.3015792","DOIUrl":"https://doi.org/10.1145/3015783.3015792","url":null,"abstract":"Computer interfaces traditionally depend on visual feedback to provide information to users, with large, high-resolution screens the norm. Other sensory modalities, such as haptics and audio, have great potential to enrich the interaction between user and device to enable new types of interaction for new user groups in new contexts. This chapter provides an overview of research in the use of these non-visual modalities for interaction, showing how new output modalities can be used in the user interface to different devices. The modalities that will be discussed include: \u0000 \u0000Haptics: tactons (vibrotactile feedback), thermal (warming and cooling feedback), force feedback, and deformable devices; \u0000 \u0000Non-Speech Audio: auditory icons, Earcons, musicons, sonification, and spatial audio output. \u0000 \u0000One motivation for using multiple modalities in a user interface is that interaction can be distributed across the different senses or control capabilities of the person using it. If one modality is fully utilized or unavailable (e.g., due to sensory or situational impairment), then another can be exploited to ensure the interaction succeeds. For example, when walking and using a mobile phone, a user needs to focus their visual attention on the environment to avoid bumping into other people. A complex visual interface on the phone may make this difficult. However, haptic or audio feedback would allow them to use their phone and navigate the world at the same time. \u0000 \u0000This chapter does not present background on multisensory perception and multimodal action, but for insights on that topic see Chapter 2. Chapter 3 also specifically discuss multisensory haptic interaction and the process of designing for it. As a complement, this chapter presents a range of applications where multimodal feedback that involves haptics or non-speech audio can provide usability benefits, motivated by Wickens' Multiple Resources Theory [Wickens 2002]. The premise of this theory is that tasks can be performed better and with fewer cognitive resources when they are distributed across modalities. For example, when driving, which is a largely visual task, route guidance is better presented through sound rather than a visual display, as that would compete with the driving for visual cognitive resources. Making calls or texting while driving, both manual tasks, would be more difficult to perform compared to voice dialing, as speech and manual input involve different modalities. For user interface design, it is important to distribute different tasks across modalities to ensure the user is not overloaded so that interaction can succeed.","PeriodicalId":222911,"journal":{"name":"The Handbook of Multimodal-Multisensor Interfaces, Volume 1","volume":"19 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-04-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124754342","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Multimodal speech and pen interfaces","authors":"Philip R. Cohen, S. Oviatt","doi":"10.1145/3015783.3015795","DOIUrl":"https://doi.org/10.1145/3015783.3015795","url":null,"abstract":"This chapter describes interfaces that enable users to combine digital pen and speech input for interacting with computing systems. Such interfaces promise natural and efficient interaction, taking advantage of skills that users have developed over many years. Many applications for such systems have been explored, such as speech and pen systems for computer-aided design (CAD), with which an architect can sketch to create and position entities while speaking information about them. For instance, a user could draw a hardwood floor outline while saying \"threefourths inch thick heart pine.\" In response, the CAD system would create a floor of the correct shape, thickness, and materials, while also updating the list of materials to purchase for the job. Then the user could touch the floor and say \"finish with polyurethane.\" The user of such a system could concentrate on creating the planned building, without interrupting their concentration to navigate a complex interface menu system. In fact, multimodal CAD systems like Think3 are preferred by users, and have been documented to significantly increase their productivity by speeding up interaction 23% [Engineer Live 2013, Price 2004]. \u0000 \u0000This chapter will discuss how speech and pen multimodal systems have been built, and also how well they have performed. By pen input we include such devices as light pens, styluses, wireless digital pens, and digital pens that can write on paper while either storing digital data, or streaming it to a receiver [Anoto 2016]. We will also occasionally refer to other devices that can, like digital pens, provide a continuous stream of coordinates---such as tracked laser pointers, finger input on touch-screens, and the ubiquitous mouse. Pen input devices can be used for a number of communicative functions, such as handwriting letters and numbers, drawing symbols, sketching diagrams or shapes, pointing, or gesturing (e.g., drawing an arrow to scroll a map). See the Glossary for defined terms. \u0000 \u0000This chapter begins by discussing users' multimodal speech and pen interaction patterns, and the documented advantages of this type of multimodal system (Section 10.2). Section 10.3 describes the simulation infrastructure that's ideally required for prototyping new systems, and the process of collecting multimodal data resources. In terms of system development, Sections 10.4 and 10.5 outline general signal processing and information flow, and major architectural components. Section 10.6 describes implemented approaches to multimodal fusion and semantic integration. Section 10.7 presents examples of multimodal speech and pen systems, some of which are commercial applications [Tumuluri 2017], with the Sketch-Thru-Plan system provided as a walk-through case study. The chapter concludes with Section 10.8 by discussing future directions for research and development. As an aid to comprehension, readers are referred to the Glossary for newly introduced terms throughout the chapter, and also t","PeriodicalId":222911,"journal":{"name":"The Handbook of Multimodal-Multisensor Interfaces, Volume 1","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-04-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126444028","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}