Dario Luipers, Nicolas Kaulen, Oliver Chojnowski, S. Schneider, A. Richert, S. Jeschke
{"title":"Robot Control Using Model-Based Reinforcement Learning With Inverse Kinematics","authors":"Dario Luipers, Nicolas Kaulen, Oliver Chojnowski, S. Schneider, A. Richert, S. Jeschke","doi":"10.1109/ICDL53763.2022.9962215","DOIUrl":"https://doi.org/10.1109/ICDL53763.2022.9962215","url":null,"abstract":"This work investigates the complications of robotic learning using reinforcement learning (RL). While RL has enormous potential for solving complex tasks its major caveat is the computation cost- and time-intensive training procedure. This work aims to address this issue by introducing a humanlike thinking and acting paradigm to a RL approach. It utilizes model-based deep RL for planning (think) coupled with inverse kinematics (IK) for the execution of actions (act). The approach was developed and tested using a Franka Emika Panda robot model in a simulated environment using the PyBullet physics engine Bullet. It was tested on three different simulated tasks and then compared to the conventional method using RL-only to learn the same tasks. The results show that the RL algorithm with IK converges significantly faster and with higher quality than the applied conventional approach, achieving 98%, 99% and 98% success rates for tasks 1-3 respectively. This work verifies its benefit for use of RL-IK with multi-joint robots.","PeriodicalId":274171,"journal":{"name":"2022 IEEE International Conference on Development and Learning (ICDL)","volume":"27 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115369965","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Dominik Mattern, Francisco M. López, M. Ernst, A. Aubret, J. Triesch
{"title":"MIMo: A Multi-Modal Infant Model for Studying Cognitive Development in Humans and AIs","authors":"Dominik Mattern, Francisco M. López, M. Ernst, A. Aubret, J. Triesch","doi":"10.1109/ICDL53763.2022.9962192","DOIUrl":"https://doi.org/10.1109/ICDL53763.2022.9962192","url":null,"abstract":"A central challenge in the early cognitive development of humans is making sense of the rich multimodal experiences originating from interactions with the physical world. AIs that learn in an autonomous and open-ended fashion based on multimodal sensory input face a similar challenge. To study such development and learning in silico, we have created MIMo, a multimodal infant model. MIMo’s body is modeled after an 18-month-old child and features binocular vision, a vestibular system, proprioception, and touch perception through a full-body virtual skin. MIMo is an open-source research platform based on the MuJoCo physics engine for constructing computational models of human cognitive development as well as studying open-ended autonomous learning in AI. We describe the design and interfaces of MIMo and provide examples illustrating its use.","PeriodicalId":274171,"journal":{"name":"2022 IEEE International Conference on Development and Learning (ICDL)","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132488235","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Learning Intrinsically Motivated Transition Models for Autonomous Systems","authors":"Khoshrav Doctor, Hia Ghosh, R. Grupen","doi":"10.1109/ICDL53763.2022.9962188","DOIUrl":"https://doi.org/10.1109/ICDL53763.2022.9962188","url":null,"abstract":"To support long-term autonomy and rational decision making, robotic systems should be risk aware and actively maintain the fidelity of critical state information. This is particularly difficult in natural environments that are dynamic, noisy, and partially observable. To support autonomy, predictive probabilistic models of robot-object interaction can be used to guide the agent toward rewarding and controllable outcomes with high probability while avoiding undesired states and allowing the agent to be aware of the amount of risk associated with acting. In this paper, we propose an intrinsically motivated learning technique to model probabilistic transition functions in a manner that is task-independent and sample efficient. We model them as Aspect Transition Graphs (ATGs)—a state-dependent control roadmap that depends on transition probability functions grounded in the sensory and motor resources of the robot. Experimental data that changes the relative perspective of an actively-controlled RGB-D camera is used to train empirical models of the transition probability functions. Our experiments demonstrate that the transition function of the underlying Partially Observable Markov Decision Process (POMDP) can be acquired efficiently using intrinsically motivated structure learning approach.","PeriodicalId":274171,"journal":{"name":"2022 IEEE International Conference on Development and Learning (ICDL)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131042339","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"The Role of the Caregiver’s Responsiveness in Affect-Grounded Language Learning by a Robot: Architecture and First Experiments","authors":"Zakaria Lemhaouri, Laura Cohen, L. Cañamero","doi":"10.1109/ICDL53763.2022.9962197","DOIUrl":"https://doi.org/10.1109/ICDL53763.2022.9962197","url":null,"abstract":"Most computational models of language development adopt a passive-learner view on language learning, and disregard the important role that motivation and affect play in the development of communication. In this paper, we present a motivation-grounded, active learning robot model of language acquisition that relies on social interaction with a caregiver. The robot learns multiple associations—between words and internal states, and between the latter and perceived objects–allowing it to have a “meaning potential” of the acquired language, which is in line with the functionalist view of language theory. We evaluate the model experimentally in different environments and with different levels of caregiver’s responsiveness to study the impact of external factors on language acquisition.","PeriodicalId":274171,"journal":{"name":"2022 IEEE International Conference on Development and Learning (ICDL)","volume":"78 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115012324","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Learning to reach to own body from spontaneous self-touch using a generative model","authors":"Valentin Marcel, J. O'Regan, M. Hoffmann","doi":"10.1109/ICDL53763.2022.9962186","DOIUrl":"https://doi.org/10.1109/ICDL53763.2022.9962186","url":null,"abstract":"When leaving the aquatic constrained environment of the womb, newborns are thrown into the world with essentially new laws and regularities that govern their interactions with the environment. Here, we study how spontaneous self-contacts can provide material for learning implicit models of the body and its action possibilities in the environment. Specifically, we investigate the space of only somatosensory (tactile and proprioceptive) activations during self-touch configurations in a simple model agent. Using biologically motivated overlapping receptive fields in these modalities, a variational autoencoder (VAE) in a denoising framework is trained on these inputs. The denoising properties of the VAE can be exploited to fill in the missing information. In particular, if tactile stimulation is provided on a single body part, the model provides a configuration that is closer to a previously experienced self-contact configuration. Iterative passes through the VAE reconstructions create a control loop that brings about reaching for stimuli on the body. Furthermore, due to the generative properties of the model, previously unsampled proprioceptive-tactile configurations can also be achieved. In the future, we will seek a closer comparison with empirical data on the kinematics of spontaneous self-touch in infants and the results of reaching for stimuli on the body.","PeriodicalId":274171,"journal":{"name":"2022 IEEE International Conference on Development and Learning (ICDL)","volume":"20 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114375531","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Multi-scale analysis of vocal coordination in infant-caregiver daily interaction","authors":"Jiarui Li, M. Casillas, S. Tsuji, Y. Nagai","doi":"10.1109/ICDL53763.2022.9962234","DOIUrl":"https://doi.org/10.1109/ICDL53763.2022.9962234","url":null,"abstract":"Infants participate in vocal coordination with others early in their life, even before they can rely on linguistic cues. They react sensitively to caregiver vocalizations, for instance, by imitating the caregiver and/or modulating their own vocalizations. When talking to an infant, caregivers also modulate their vocalizations, e.g., talk more slowly or with exaggerated prosody, which might attract infants’ attention and increase the clarity of vocal information. However, it is still unclear to what extent both parties’ vocal modifications dynamically influence each other. In this study, we investigate infants’ and caregivers’ vocal coordination in their daily interactions by applying multi-scale analysis on a global scale (i.e., a day), a middle scale (i.e., a conversational block), and a local scale (i.e., a turn). The day-long auditory recording data of nine infants, ages two months to three years, and their caregivers were analyzed. The results revealed that infants’ and caregivers’ vocalizations are differently coordinated on each timescale. On a global scale, infants and mothers react sensitively to each other’s vocalizations. Their conversation length varies across a day with a decreasing tendency. On a middle scale, infant-caregivers’ prosodic alignments increase over multiple turns in a conversation, indicating a continuous influence between them. Finally, more fine-grained analyses found that pitch-related features and pitch contours are aligned in each turn. The multi-scale analysis reveals the complexity of infant-caregiver interaction in the natural social environment, which inspires us to investigate the benefits of alignment in infants’ language learning at different timescales.","PeriodicalId":274171,"journal":{"name":"2022 IEEE International Conference on Development and Learning (ICDL)","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117002183","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Autonomous learning of multiple curricula with non-stationary interdependencies*","authors":"A. Romero, G. Baldassarre, R. Duro, V. Santucci","doi":"10.1109/ICDL53763.2022.9962200","DOIUrl":"https://doi.org/10.1109/ICDL53763.2022.9962200","url":null,"abstract":"Autonomous open-ended learning is a relevant approach in machine learning and robotics, allowing artificial agents to acquire a wide repertoire of goals and motor skills without the necessity of specific assignments. Leveraging intrinsic motivations, different works have developed systems that can autonomously allocate training time amongst different goals to maximise their overall competence. However, only few works in the field of intrinsically motivated open-ended learning focus on scenarios where goals have interdependent relations, and even fewer tackle scenarios involving non-stationary interdependencies. Building on previous works, we propose a new hierarchical architecture (H-GRAIL) that selects its own goals on the basis of intrinsic motivations and treats curriculum learning of interdependent tasks as a Markov Decision Process. Moreover, we provide H-GRAIL with a novel mechanism that allows the system to self-regulate its exploratory behaviour and cope with the non-stationarity of the dependencies between goals. The system is tested in a simulated and real robotic environment with different experimental scenarios involving interdependent tasks.","PeriodicalId":274171,"journal":{"name":"2022 IEEE International Conference on Development and Learning (ICDL)","volume":"2016 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127373822","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Visuo-Motor Remapping for 3D, 6D and Tool-Use Reach using Gain-Field Networks","authors":"Xiaodan Chen, Alexandre Pitti","doi":"10.1109/ICDL53763.2022.9962219","DOIUrl":"https://doi.org/10.1109/ICDL53763.2022.9962219","url":null,"abstract":"Reaching and grasping objects in 3D is still a challenging task in robotics because they have to be done in an integrated fashion, as it is for tool-use or during imitation with a human partner. The visuo-motor networks in the human brain exploit a neural mechanism known as gain-field modulation to adapt different circuits together with respect to the task and for parsimony purpose. In this paper, we show how gain-field neural networks achieve the learning of visuo-motor cells sensitive to the 3D direction of the arm motion (3D reaching), to the 3D reaching + 3D orientation of the hand (6D reaching) and to the 3D direction of tool tip (tool-use reaching) when this new information is added to the network. Experiments on robotic simulations demonstrate the accuracy of control and the efficient remapping to the new coordinate system.","PeriodicalId":274171,"journal":{"name":"2022 IEEE International Conference on Development and Learning (ICDL)","volume":"36 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114449857","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Sara Marques-Villarroya, Juan José Gamboa-Montero, A. Bernardino, Marcos Maroto-Gómez, J. C. Castillo, M. Salichs
{"title":"Real-time Engagement Detection from Facial Features","authors":"Sara Marques-Villarroya, Juan José Gamboa-Montero, A. Bernardino, Marcos Maroto-Gómez, J. C. Castillo, M. Salichs","doi":"10.1109/ICDL53763.2022.9962228","DOIUrl":"https://doi.org/10.1109/ICDL53763.2022.9962228","url":null,"abstract":"Nowadays, engagement detection plays an essential role in e-learning education and robotics. In the field of human-agent interaction, it is of great interest to know the attitude of the human peer towards the interaction so that the agent can react accordingly. The goal of this paper is to develop an automatic real-time engagement recognition system using a combination of non-verbal features (gaze direction, head position, facial expression and distance between users) extracted using computer vision techniques. Our system uses a machine learning model based on Random Forest and achieves 86% accuracy, improving the results of the state-of-the-art methods by 22.2% in engagement level detection accuracy on the Daisee dataset. Furthermore, using an RGB camera, the system can detect the level of user engagement in real-time and classify it into four levels of intensity.","PeriodicalId":274171,"journal":{"name":"2022 IEEE International Conference on Development and Learning (ICDL)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129186684","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
M. Kirtay, Erhan Öztop, A. Kuhlen, M. Asada, V. Hafner
{"title":"Forming robot trust in heterogeneous agents during a multimodal interactive game","authors":"M. Kirtay, Erhan Öztop, A. Kuhlen, M. Asada, V. Hafner","doi":"10.1109/ICDL53763.2022.9962212","DOIUrl":"https://doi.org/10.1109/ICDL53763.2022.9962212","url":null,"abstract":"This study presents a robot trust model based on cognitive load that uses multimodal cues in a learning setting to assess the trustworthiness of heterogeneous interaction partners. As a test-bed, we designed an interactive task where a small humanoid robot, Nao, is asked to perform a sequential audio-visual pattern recall task while minimizing its cognitive load by receiving help from its interaction partner, either a robot, Pepper, or a human. The partner displayed one of three guiding strategies, reliable, unreliable, or random. The robot is equipped with two cognitive modules: a multimodal auto-associative memory and an internal reward module. The former represents the multimodal cognitive processing of the robot and allows a ‘cognitive load’ or ‘cost’ to be assigned to the processing that takes place, while the latter converts the cognitive processing cost to an internal reward signal that drives the cost-based behavior learning. Here, the robot asks for help from its interaction partner when its action leads to a high cognitive load. Then the robot receives an action suggestion from the partner and follows it. After performing interactive experiments with each partner, the robot uses the cognitive load yielded during the interaction to assess the trustworthiness of the partners –i.e., it associates high trustworthiness with low cognitive load. We then give a free choice to the robot to select the trustworthy interaction partner to perform the next task. Our results show that, overall, the robot selects partners with reliable guiding strategies. Moreover, the robot’s ability to identify a trustworthy partner was unaffected by whether the partner was a human or a robot.","PeriodicalId":274171,"journal":{"name":"2022 IEEE International Conference on Development and Learning (ICDL)","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133581742","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}