Zahra J. Muhsin, Rami Qahwaji, Faruque Ghanchi, Majid Al-Taee
{"title":"Review of substitutive assistive tools and technologies for people with visual impairments: recent advancements and prospects","authors":"Zahra J. Muhsin, Rami Qahwaji, Faruque Ghanchi, Majid Al-Taee","doi":"10.1007/s12193-023-00427-4","DOIUrl":"https://doi.org/10.1007/s12193-023-00427-4","url":null,"abstract":"<p>The development of many tools and technologies for people with visual impairment has become a major priority in the field of assistive technology research. However, many of these technology advancements have limitations in terms of the human aspects of the user experience (e.g., usability, learnability, and time to user adaptation) as well as difficulties in translating research prototypes into production. Also, there was no clear distinction between the assistive aids of adults and children, as well as between “partial impairment” and “total blindness”. As a result of these limitations, the produced aids have not gained much popularity and the intended users are still hesitant to utilise them. This paper presents a comprehensive review of substitutive interventions that aid in adapting to vision loss, centred on laboratory research studies to assess user-system interaction and system validation. Depending on the primary cueing feedback signal offered to the user, these technology aids are categorized as visual, haptics, or auditory-based aids. The context of use, cueing feedback signals, and participation of visually impaired people in the evaluation are all considered while discussing these aids. Based on the findings, a set of recommendations is suggested to assist the scientific community in addressing persisting challenges and restrictions faced by both the totally blind and partially sighted people.</p>","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":null,"pages":null},"PeriodicalIF":2.9,"publicationDate":"2023-12-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138742323","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Augmented reality and deep learning based system for assisting assembly process","authors":"","doi":"10.1007/s12193-023-00428-3","DOIUrl":"https://doi.org/10.1007/s12193-023-00428-3","url":null,"abstract":"<h3>Abstract</h3> <p>In Industry 4.0, manufacturing entails a rapid change in customer demands which leads to mass customization. The variation in customer requirements leads to small batch sizes and several process variations. Assembly task is one of most important steps in any manufacturing process. A factory floor worker often needs a guidance system due to variations in product or process, to assist them in assembly task. Existing Augmented Reality (AR) based systems use markers for each assembly component for detection which is time consuming and laborious. This paper proposed utilizing state-of-the-art deep learning based object detection technique and employed a regression based mapping technique to obtain the 3D locations of assembly components. Automatic detection of machine parts was followed by a multimodal interface involving both eye gaze and hand tracking to guide the manual assembly process. We proposed eye cursor to guide the user through the task and utilized fingertip distances along with object sizes to detect any error committed during the task. We analyzed the proposed mapping method and found that the mean mapping error was 1.842 cm. We also investigated the effectiveness of the proposed multimodal user interface by conducting two user studies. The first study indicated that the current interface design with eye cursor enabled participants to perform the task significantly faster compared to the interface without eye cursor. The shop floor workers during the second user study reported that the proposed guidance system was comprehendible and easy to use to complete the assembly task. Results showed that the proposed guidance system enabled 11 end users to finish the assembly of one pneumatic cylinder within 55 s with average TLX score less than 25 in a scale of 100 and Cronbach alpha score of 0.8 indicating convergence of learning experience.</p>","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":null,"pages":null},"PeriodicalIF":2.9,"publicationDate":"2023-12-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138684924","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Beatrice Biancardi, Maurizio Mancini, Brian Ravenet, Giovanna Varni
{"title":"Modelling the “transactive memory system” in multimodal multiparty interactions","authors":"Beatrice Biancardi, Maurizio Mancini, Brian Ravenet, Giovanna Varni","doi":"10.1007/s12193-023-00426-5","DOIUrl":"https://doi.org/10.1007/s12193-023-00426-5","url":null,"abstract":"Abstract Transactive memory system (TMS) is a team emergent state representing the knowledge of each member about “who knows what” in a team performing a joint task. We present a study to show how the three TMS dimensions Credibility, Specialisation, Coordination, can be modelled as a linear combination of the nonverbal multimodal features displayed by the team performing the joint task. Results indicate that, to some extent, the three dimensions of TMS can be expressed as a linear combination of nonverbal multimodal features. Moreover, the higher the number of modalities (audio, movement, spatial), the better the modelling. Results could be used in future work to design human-centered computing applications able to automatically estimate TMS from teams’ behavioural patterns, to provide feedback and help teams’ interactions.","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135042028","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Model-based sonification based on the impulse pattern formulation","authors":"Simon Linke, Rolf Bader, Robert Mores","doi":"10.1007/s12193-023-00423-8","DOIUrl":"https://doi.org/10.1007/s12193-023-00423-8","url":null,"abstract":"Abstract The most common strategy for interactive sonification is parameter mapping sonification, where sensed or defined data is pre-processed and then used to control one or more variables in a signal processing chain. A well-known but rarely used alternative is model-based sonification, where data is fed into a physical or conceptual model that generates or modifies sound. In this paper, we suggest the Impulse Pattern Formulation (IPF) as a model-based sonification strategy. The IPF can model natural systems and interactions, like the sound production of musical instruments, the reverberation in rooms, and human synchronization to a rhythm. Hence, the IPF has the potential to be easy to interpret and intuitive to interact with. Experiment results show that the IPF is able to produce an intuitively interpretable, natural zero, i.e., a coordinate origin. Coordinate origins are necessary to sonify both polarities of a dimension as well as absolute magnitudes.","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135679078","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Tim Ziemer, Sara Lenzi, Niklas Rönnberg, Thomas Hermann, Roberto Bresin
{"title":"Introduction to the special issue on design and perception of interactive sonification","authors":"Tim Ziemer, Sara Lenzi, Niklas Rönnberg, Thomas Hermann, Roberto Bresin","doi":"10.1007/s12193-023-00425-6","DOIUrl":"https://doi.org/10.1007/s12193-023-00425-6","url":null,"abstract":"","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-11-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135933758","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Correction to: PepperOSC: enabling interactive sonification of a robot’s expressive movement","authors":"Adrian B. Latupeirissa, Roberto Bresin","doi":"10.1007/s12193-023-00424-7","DOIUrl":"https://doi.org/10.1007/s12193-023-00424-7","url":null,"abstract":"","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136102527","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Three-dimensional sonification as a surgical guidance tool","authors":"Ziemer, Tim","doi":"10.1007/s12193-023-00422-9","DOIUrl":"https://doi.org/10.1007/s12193-023-00422-9","url":null,"abstract":"Interactive Sonification is a well-known guidance method in navigation tasks. Researchers have repeatedly suggested the use of interactive sonification in neuronavigation and image-guided surgery. The hope is to reduce clinicians' cognitive load through a relief of the visual channel, while preserving the precision provided through image guidance. In this paper, we present a surgical use case, simulating a craniotomy preparation with a skull phantom. Through auditory, visual, and audiovisual guidance, non-clinicians successfully find targets on a skull that provides hardly any visual or haptic landmarks. The results show that interactive sonification enables novice users to navigate through three-dimensional space with a high precision. The precision along the depth axis is highest in the audiovisual guidance mode, but adding audio leads to higher durations and longer motion trajectories.","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-10-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136233550","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Weitao Jiang, Bingxin Zhang, Ruiqi Sun, Dong Zhang, Shan Hu
{"title":"A study on the attention of people with low vision to accessibility guidance signs","authors":"Weitao Jiang, Bingxin Zhang, Ruiqi Sun, Dong Zhang, Shan Hu","doi":"10.1007/s12193-023-00417-6","DOIUrl":"https://doi.org/10.1007/s12193-023-00417-6","url":null,"abstract":"","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-10-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134908962","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Mariana Seiça, Licínio Roque, Pedro Martins, F. Amílcar Cardoso
{"title":"An interdisciplinary journey towards an aesthetics of sonification experience","authors":"Mariana Seiça, Licínio Roque, Pedro Martins, F. Amílcar Cardoso","doi":"10.1007/s12193-023-00416-7","DOIUrl":"https://doi.org/10.1007/s12193-023-00416-7","url":null,"abstract":"Abstract The aesthetic dimension has been proposed as a potential expansion of sonification design, creating listening pieces that reach the goal of effective data communication. However, current views of aesthetics still aim at optimising mapping criteria to convey the ‘right meaning’, maintaining a mostly functional view on what is considered a successful sonification. This paper proposes an interdisciplinary approach to the aesthetics of sonification experience, grounded on theoretical foundations from phenomenology of interaction, post-phenomenology, cross-cultural studies, acoustic ecology and deep listening. From this journey, we present the following design insights: (1) the design of sonifications becomes a design for experience, (2) co-designed during the interaction with each participant; (3) the sonification artefact gains a mediating role that guides the participant’s intentions in the sonification space; (4) the aesthetics of a sonification experience generates a multistable phenomenon, offering new opportunities to experience multiple perspectives over data; (5) the interaction between human participants and the sonic emanations compose a dialogic space. A call for action to reframe the sonification field into novel design spaces is now open, with aesthetics gaining a transformational role in sonification design and interaction.","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-10-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135463803","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Comparing alternative modalities in the context of multimodal human–robot interaction","authors":"Suprakas Saren, Abhishek Mukhopadhyay, Debasish Ghose, Pradipta Biswas","doi":"10.1007/s12193-023-00421-w","DOIUrl":"https://doi.org/10.1007/s12193-023-00421-w","url":null,"abstract":"","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-10-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135730017","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}