The Handbook of Multimodal-Multisensor Interfaces, Volume 1最新文献

{"title":"Multimodal technologies for seniors: challenges and opportunities","authors":"Cosmin Munteanu, A. A. Salah","doi":"10.1145/3015783.3015793","DOIUrl":"https://doi.org/10.1145/3015783.3015793","url":null,"abstract":"This chapter discusses interactive technologies in the service of seniors. Adults over 65 form one of the largest and most rapidly growing user groups in the industrialized society. Interactive technologies have been steadily improving in their ability to provide practical support for these users and their daily challenges. These applications range from supporting low-literacy adults with mobile touch and spoken language interfaces [Munteanu et al. 2014], to interactive robotic fitness coaches for seniors [Gorer et al. 2016]. The goal of this chapter is to present an in-depth analysis of the needs of seniors, as well as to investigate how multimodal and multisensor technologies are used to address these needs. \u0000 \u0000In Section 8.2 of this chapter, we structure the needs of older adults according to a hierarchy of human needs that emphasizes self-actualization [Maslow 1954]. Maslow's theoretical prism helps us to identify a broad spectrum of potential opportunities for researchers and practitioners who design multimodal-multisensor applications. We present a number of example applications in Section 8.3. These examples are illustrative of how interactive technologies, especially ones that leverage multiple modalities (e.g., speech, gestures, touch) and sensor data (e.g., location and accelerometers available on consumer phones) can provide a basis for meeting the needs of older adults. Section 8.4 briefly surveys future directions of multimodal-multisensor research that has potential for developing valuable new 320 Chapter 8 Multimodal Technologies for Seniors: Challenges and Opportunities assistive living technologies.1 Section 8.5 is a discussion of both design and implementation challenges, followed with conclusions in Section 8.6, and a list of supplementary Focus Questions to aid in further understanding of essential content. We hope that this broad overview will encourage both new and established researchers to explore how current or future multimodal and multisensor interfaces can better support one of the largest but most under-represented demographic groups.","PeriodicalId":222911,"journal":{"name":"The Handbook of Multimodal-Multisensor Interfaces, Volume 1","volume":"4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-04-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114653981","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Perspectives on learning with multimodal technology","authors":"K. James, James C. Lester, Daniel L. Schwartz, Katherine M. Cheng, S. Oviatt","doi":"10.1145/3015783.3015798","DOIUrl":"https://doi.org/10.1145/3015783.3015798","url":null,"abstract":"To set the stage for this multidisciplinary discussion among experts on the challenging topic of learning with multimodal technology, weasksomebasic questions: \u0000 \u0000• What have neuroscience, cognitive and learning sciences, and humancomputer interaction findings taught us about how humans learn? \u0000 \u0000• What are the implications for designing more effective educational technologies, in particular ones that leverage emerging multimodal-multisensor capabilities? \u0000 \u0000Computer technologies are becoming an increasingly influential set of tools in a student's classroom experience. However, large-scale assessments conducted on laptop initiatives and other technology programs have not always yielded significantly improved student performance on standardized achievement tests or cognitive measures. The purpose of this chapter is to discuss what role multimodalmultisensor technologies potentially could play in improving support for learning and education. The expert neuroscience, cognitive and learning sciences, and computer science discussants involved in this exchange consider what we know about how multisensory-multimodal learning occurs, and the implications for how we could develop multimodal-multisensor interfaces that more effectively stimulate thinking and learning than past computer interfaces.","PeriodicalId":222911,"journal":{"name":"The Handbook of Multimodal-Multisensor Interfaces, Volume 1","volume":"78 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-04-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121130514","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Using cognitive models to understand multimodal processes: the case for speech and gesture production","authors":"S. Kopp, K. Bergmann","doi":"10.1145/3015783.3015791","DOIUrl":"https://doi.org/10.1145/3015783.3015791","url":null,"abstract":"Multimodal behavior has been studied for a long time and in many fields, e.g., in psychology, linguistics, communication studies, education, and ergonomics. One of the main motivations has been to allow humans to use technical systems intuitively, in a way that resembles and fosters human users' natural way of interacting and thinking [Oviatt 2013]. This has sparked early work on multimodal human-computer interfaces, including recent approaches to recognize communicative behavior and even subtle multimodal cues by computer systems. Those approaches, for the most part, rest on machine learning techniques applied to large sets of behavioral data. As datasets grow larger in size and coverage, and computational power increases, suitable data-driven techniques are able to detect correlational behavior patterns that support answering questions like which feature( s) to take into account or how to recognize them in specific contexts. However, natural multimodal interaction in humans entails a plethora of behavioral variations and intricacies (e.g., when to act unimodally vs. multimodally, with which specific behaviors or multi-level coordination between them). Possible underlying patterns are hard to detect, even in large datasets, and often such variations are attributed to context-dependencies or individual differences. How they come about is still hard to explain at the behavioral level. \u0000 \u0000One additional level of explanation that can help to deepen our understanding, and to establish systematic and generalized accounts, involves cognitive processes that lead to a particular multimodal response in a specific situation (e.g., see Chapter 2; [James et al. 2017]). A prominent example is the concept of cognitive load or \"mental workload.\" Many behavioral variations in spoken language have been meaningfully explained in terms of heightened or lowered cognitive load of the speaker. For example, under high cognitive load speakers are found to speak more slowly, to produce more silent or filled pauses, and to utter more repetitions [Jameson et al. 2010]. Likewise, human users distribute information across multiple modalities in order to manage their cognitive limits [Oviatt et al. 2004, Chen et al. 2012]. In studies with elementary school children as well as adults, active manual gesturing was demonstrated to improve memory during a task that required explaining math solutions [Goldin-Meadow et al. 2001]. This effect of gesturing increased with higher task difficulty. Such behavioral phenomena are commonly explained based on cognitive concepts like cognitive load and, further, underlying processes like modality-specific working memories [Baddeley 1992] or competition for cognitive resources [Wickens et al. 1983]. \u0000 \u0000Cognitive theories also provide valuable hints as to how to design multimodal human-machine interaction. To continue with the above example, it has been suggested that certain multimodal interfaces help users to minimize their cognitive load and ","PeriodicalId":222911,"journal":{"name":"The Handbook of Multimodal-Multisensor Interfaces, Volume 1","volume":"34 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-04-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126963645","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}