{"title":"Assemble like human: A multi-level imitation model learning human perception-decision-operation skills for robot automatic assembly tasks","authors":"Hubo Chu , Tie Zhang , Yanbiao Zou , Hanlei Sun","doi":"10.1016/j.rcim.2024.102907","DOIUrl":"10.1016/j.rcim.2024.102907","url":null,"abstract":"<div><div>Robot automatic peg-in-hole assembly is a challenging task. Human perception-decision-operation skills play an irreplaceable role in precise assembly. In this paper, human assembly skills are modeled as a multi-level Markov decision process, and a multi-level imitation model is proposed to learn human assembly skills from demonstrations. Specifically, to learn human skills of identifying assembly phases based on perception signals, a perception-decision model based on parallel encoding Gaussian mixture model and decision correction module is proposed, and a new loss function combining reconstruction error and likelihood is designed. To accurately learn human operation skills from imprecise demonstrations, a decision-operation model combining mixture density networks and energy-based models is proposed. Based on the multi-level imitation model, a robot assembly controller is designed to drive robots to assemble like humans. Comparative experiments and peg-in-hole assembly experiments with four clearances indicate that the proposed method can more accurately learn human perception-decision-operation skills and achieve a better assembly effect than the advanced methods.</div></div>","PeriodicalId":21452,"journal":{"name":"Robotics and Computer-integrated Manufacturing","volume":"93 ","pages":"Article 102907"},"PeriodicalIF":9.1,"publicationDate":"2024-12-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142788935","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Integrated optimization of storage space allocation and crane scheduling in automated storage and retrieval systems","authors":"Wenbin Zhang , Zhiyun Deng , Chunjiang Zhang , Weiming Shen","doi":"10.1016/j.rcim.2024.102918","DOIUrl":"10.1016/j.rcim.2024.102918","url":null,"abstract":"<div><div>This paper addresses the challenge of integrated optimization for storage space allocation and crane scheduling in automated storage and retrieval systems. The problem encompasses tasks such as assigning storage/retrieval requests, allocating storage spaces, and planning crane routes within each operation cycle. To tackle this, we introduce a multi-layer adaptive length coding method to effectively map the solution space to the problem space. Employing a coevolutionary framework, we decompose and process the integrated optimization problem, further optimize it with a hybrid genetic algorithm. Numerical experiments across a wide range of scenarios are conducted to evaluate the algorithm’s performance under varying request sizes and crane capacities. The introduction of the coevolutionary framework improves optimization by up to 14.78%, with an average improvement of 34.09% compared to the method currently used in the company. In addition, we introduce a novel optimization metric, termed potential energy consumption, designed to enhance system energy efficiency. Comparative analysis against metrics like makespan reveals the superiority of our proposed approach in terms of coverage and optimality, particularly in large-scale scenarios. The combined implementation of integrated optimization and the new evaluation metric leads to substantial energy cost savings for real-world automated storage and retrieval systems.</div></div>","PeriodicalId":21452,"journal":{"name":"Robotics and Computer-integrated Manufacturing","volume":"93 ","pages":"Article 102918"},"PeriodicalIF":9.1,"publicationDate":"2024-12-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142788896","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Maxiao Hou , Jianghai Shi , Xiaoman Lin , Weijun Tian , Ying Xue , Shening Qiao , Hongrui Cao
{"title":"Optimization of robot posture and spindle speed in robotic milling","authors":"Maxiao Hou , Jianghai Shi , Xiaoman Lin , Weijun Tian , Ying Xue , Shening Qiao , Hongrui Cao","doi":"10.1016/j.rcim.2024.102921","DOIUrl":"10.1016/j.rcim.2024.102921","url":null,"abstract":"<div><div>In robotic machining, most of the existing research has been done to improve the machining performance by optimizing the robot posture. However, spindle speed also plays an important role in improving machining performance. In this paper, a robot posture and spindle speed optimization method is proposed to improve the machining performance in robotic milling. First, the frequency response function is measured by the modal test. Based on the measured frequency response function, the frequency response function of the remaining redundant angle and machining position is predicted by the Gaussian process regression model. Next, the profile error is obtained based on the predicted frequency response function and the simulated milling force. The machining performance index in robotic milling is given on the basis of the profile error. Then, the Intelligible-in-time Logics algorithm (ILA) is introduced to find the optimal robot posture and spindle speed based on the machining performance index. In addition, the machining performance index threshold is specified to reduce the redundant angle and spindle speed variation for practical requirements. Finally, the optimized robot posture and spindle speed can further improve the machining performance in robotic milling, which is verified by the robotic milling experiment. The experimental results show that the proposed method can reduce the peak acceleration by 46.82% for the same machining path.</div></div>","PeriodicalId":21452,"journal":{"name":"Robotics and Computer-integrated Manufacturing","volume":"93 ","pages":"Article 102921"},"PeriodicalIF":9.1,"publicationDate":"2024-12-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143151890","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Automating the hand layup process: On the removal of protective films with collaborative robots","authors":"Renat Kermenov , Sergi Foix , Júlia Borràs , Vincenzo Castorani , Sauro Longhi , Andrea Bonci","doi":"10.1016/j.rcim.2024.102899","DOIUrl":"10.1016/j.rcim.2024.102899","url":null,"abstract":"<div><div>This paper explores the issue of protective film removal in the hand layup process for composite parts production. The hand layup process, involving the assembly of prepreg plies onto a mold, is a skill-intensive task performed by multiple expert workers. A significant limitation of this method is its low repeatability, which impacts both the consistency and quality of the final product. The current research trend has the objective of developing autonomous or semi-autonomous layup cells to enhance process consistency, reduce production costs, and improve product quality.</div><div>Despite all this interest in bringing automation in composite manufacturing, an area left relatively unexplored is the removal of protective films from prepregs. The plies used in the hand layup process, are generally covered by those films that are removed by the workers during the manual layup activity. The manual removal of protective films from prepregs is a tedious and valueless task, which represents a bottleneck in achieving full or semi-automation of the layup process. For this reason, an autonomous or semi-autonomous cell needs to perform it to be market-relevant.</div><div>In this work, we propose a new effective method for initiating the peeling and integrate this method into a complete framework for the removal of protective films. This solution is designed to be easily integrated into a variety of existing cells. Finally, we validate our framework with an experimental proof of concept (PoC) which makes use of two collaborative robots for task execution.</div></div>","PeriodicalId":21452,"journal":{"name":"Robotics and Computer-integrated Manufacturing","volume":"93 ","pages":"Article 102899"},"PeriodicalIF":9.1,"publicationDate":"2024-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142788901","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Baicun Wang , Ci Song , Xingyu Li , Huiying Zhou , Huayong Yang , Lihui Wang
{"title":"A deep learning-enabled visual-inertial fusion method for human pose estimation in occluded human-robot collaborative assembly scenarios","authors":"Baicun Wang , Ci Song , Xingyu Li , Huiying Zhou , Huayong Yang , Lihui Wang","doi":"10.1016/j.rcim.2024.102906","DOIUrl":"10.1016/j.rcim.2024.102906","url":null,"abstract":"<div><div>In the context of human-centric smart manufacturing, human-robot collaboration (HRC) systems leverage the strengths of both humans and machines to achieve more flexible and efficient manufacturing. In particular, estimating and monitoring human motion status determines when and how the robots cooperate. However, the presence of occlusion in industrial settings seriously affects the performance of human pose estimation (HPE). Using more sensors can alleviate the occlusion issue, but it may cause additional computational costs and lower workers' comfort. To address this issue, this work proposes a visual-inertial fusion-based method for HPE in HRC, aiming to achieve accurate and robust estimation while minimizing the influence on human motion. A part-specific cross-modal fusion mechanism is designed to integrate spatial information provided by a monocular camera and six Inertial Measurement Units (IMUs). A multi-scale temporal module is developed to model the motion dependence between frames at different granularities. Our approach achieves 34.9 mm Mean Per Joint Positional Error (MPJPE) on the TotalCapture dataset and 53.9 mm on the 3DPW dataset, outperforming state-of-the-art visual-inertial fusion-based methods. Tests on a synthetic-occlusion dataset further validate the occlusion robustness of our network. Quantitative and qualitative experiments on a real assembly case verified the superiority and potential of our approach in HRC. It is expected that this work can be a reference for human motion perception in occluded HRC scenarios.</div></div>","PeriodicalId":21452,"journal":{"name":"Robotics and Computer-integrated Manufacturing","volume":"93 ","pages":"Article 102906"},"PeriodicalIF":9.1,"publicationDate":"2024-11-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142747246","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A self-imitation learning approach for scheduling evaporation and encapsulation stages of OLED display manufacturing systems","authors":"Donghun Lee , In-Beom Park , Kwanho Kim","doi":"10.1016/j.rcim.2024.102917","DOIUrl":"10.1016/j.rcim.2024.102917","url":null,"abstract":"<div><div>In modern organic light-emitting diode (OLED) manufacturing systems, scheduling is a key decision-making problem to improve productivity. In particular, the scheduling of evaporation and encapsulation stages has been confronted with complicated constraints such as job-splitting property, preventive maintenance, machine eligibility, family setups, and heterogeneous release time of jobs. To efficiently solve such complicated scheduling problems, reinforcement learning (RL) has drawn increasing attention as an alternative in recent years. Unfortunately, the performance of the RL-based scheduling methods might not be satisfactory since unexpected correlations between actions are caused by machine eligibility restrictions, making it more challenging to address the credit assignment problem. To minimize the total tardiness, this article proposes a self-imitation learning-based scheduling method in which an agent utilizes past good experiences to exploit efficient exploration. Furthermore, a novel return design is introduced to overcome the credit assignment problem by considering machine eligibility restrictions. To prove the effectiveness and efficiency of the proposed method, numerical experiments are carried out by using the datasets that simulated the real-world OLED display manufacturing systems. Experiment results demonstrate that the proposed method outperforms other baselines, including rule-based and meta-heuristics, as well as the other DRL-based method in terms of the total tardiness while reducing computation time compared to meta-heuristics.</div></div>","PeriodicalId":21452,"journal":{"name":"Robotics and Computer-integrated Manufacturing","volume":"93 ","pages":"Article 102917"},"PeriodicalIF":9.1,"publicationDate":"2024-11-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142747248","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Hepeng Ni , Tianliang Hu , Jindong Deng , Bo Chen , Shuangsheng Luo , Shuai Ji
{"title":"Digital twin-driven virtual commissioning for robotic machining enhanced by machine learning","authors":"Hepeng Ni , Tianliang Hu , Jindong Deng , Bo Chen , Shuangsheng Luo , Shuai Ji","doi":"10.1016/j.rcim.2024.102908","DOIUrl":"10.1016/j.rcim.2024.102908","url":null,"abstract":"<div><div>Robotic machining has been increasingly applied in intelligent manufacturing production lines. Compared with the traditional machine tools, commissioning for robotic machining system (RMS) is particularly important due to the low accuracy of industrial robots (IRs). Traditional site commissioning has large workload and is difficult to handle the multi-source errors. Since digital twin (DT) provides strategies for staying synchronized with the physical entities in whole lifecycle, a DT-driven virtual commissioning (VC) system for RMS is developed in this study to improve machining accuracy and reduce the difficulty of commissioning. Firstly, the framework of DT-driven VC system is designed including several function modules such as interaction, data pre-processing, DT model of RMS (RMSDT), and optimization service. Since RMSDT is the kernel of precise VC, a machine learning-enhanced RMSDT oriented to actual machining path prediction is then constructed based on a proposed joint error equivalent strategy, which can fully consider the coupled multi-source errors of machining robot. After that, a practical consistency retention method for RMSDT is proposed based on a stepwise updating strategy, where the model performance can be maintained with low updating costs. Finally, a visual VC system is developed for the experimental 6-degree of freedom robotic milling platform to verify the feasibility and effectiveness of the VC framework. Multiple experiments are also performed to test the performance of RMSDT and contour error compensation. This study has useful reference for the enterprises engaged in RMS and has positive significance for promoting the robotic machining.</div></div>","PeriodicalId":21452,"journal":{"name":"Robotics and Computer-integrated Manufacturing","volume":"93 ","pages":"Article 102908"},"PeriodicalIF":9.1,"publicationDate":"2024-11-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142747247","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Efficient tool path planning method of ball-end milling for high quality manufacturing","authors":"Hong-Yu Ma , Yi-Bo Kou , Li-Yong Shen , Chun-Ming Yuan","doi":"10.1016/j.rcim.2024.102905","DOIUrl":"10.1016/j.rcim.2024.102905","url":null,"abstract":"<div><div>Triangular mesh representation is extensively utilized in geometric design and reverse engineering. However, in the realm of high quality CNC machining, there is a notable transition from mesh to continuous surface representation for workpieces. This paper presents a novel approach to address this shift, proposing a high-precision and efficient path generation method of ball-end milling specifically designed for triangular meshes. The method integrates considerate surface fitting techniques with productive path planning strategies to optimize machining processes. The method first introduces GNURBS surface fitting adapted for CAM with normal vectors and sharp features preserving, then provides a surface segmentation strategy better suited for machining based on a weighted graph analysis, and finally presents a Fermat spirals path generation scheme with single start and end points. Experimental results and case studies are provided to illustrate and clarify our method. The results show the superior performance and effectiveness of our method concerning surface quality, sharp features, and machining time.</div></div>","PeriodicalId":21452,"journal":{"name":"Robotics and Computer-integrated Manufacturing","volume":"93 ","pages":"Article 102905"},"PeriodicalIF":9.1,"publicationDate":"2024-11-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142719940","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Yong Tao , Jiahao Wan , Yian Song , Xingyu Li , Baicun Wang , Tianmiao Wang , Yiru Wang
{"title":"A safety posture field framework for mobile manipulators based on human–robot interaction trend and platform-arm coupling motion","authors":"Yong Tao , Jiahao Wan , Yian Song , Xingyu Li , Baicun Wang , Tianmiao Wang , Yiru Wang","doi":"10.1016/j.rcim.2024.102903","DOIUrl":"10.1016/j.rcim.2024.102903","url":null,"abstract":"<div><div>Mobile manipulators are increasingly deployed in industrial settings, such as material handling and workpiece loading, where they must safely interact with humans while efficiently completing tasks. Existing motion planning methods for mobile manipulators often struggle to ensure both safety and efficiency in dynamic human-robot interaction environments. This paper proposes a Safety Posture Field framework that addresses these limitations by firstly predicting human motion trends using the improved Long Short-Term Memory neural network and applying these predictions to potential field calculations for both the mobile platform and the robotic arm. During different stages of human-robot interaction, the mobile manipulator places varying emphasis on safety and efficiency while in motion. Additionally, when the robotic arm executes operations, a platform-arm coupling motion strategy is introduced when the potential field detects risks of singularity or local optima, preventing the robotic arm from becoming unstable or failing to reach the target pose in time. This strategy enhances the system's flexibility and operational stability. Comparative experiments in simulation and real-world settings confirm the ability of the framework to maintain high safety standards while improving task efficiency, making it suitable for industrial Human-Robot Interaction applications.</div></div>","PeriodicalId":21452,"journal":{"name":"Robotics and Computer-integrated Manufacturing","volume":"93 ","pages":"Article 102903"},"PeriodicalIF":9.1,"publicationDate":"2024-11-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142702679","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Shizhong Tan, Jixiang Yang, Chengxing Wu, Han Ding
{"title":"Processing accuracy improvement of robotic ball-end milling by simultaneously optimizing tool orientation and robotic redundancy","authors":"Shizhong Tan, Jixiang Yang, Chengxing Wu, Han Ding","doi":"10.1016/j.rcim.2024.102904","DOIUrl":"10.1016/j.rcim.2024.102904","url":null,"abstract":"<div><div>Robotic ball-end milling presents advantages such as a broad workspace, cost-effectiveness, and integration with vision/force sensing, making it a promising method in machinery manufacturing. However, its low stiffness leads to deformation error that seriously affects part profile accuracy. Reducing the deformation error is an effective method to improve the machining accuracy of robotic milling. However, existing research primarily focuses on translational deformation of the robot end effector calculated using average cutting force, overlooking the effect of changes in cutting force and deformation at the tool tip. To address these limitations, an optimization model is proposed to simultaneously optimize tool orientation and redundant angle to minimize force-induced tool tip deformation errors, accounting for cutting force variations at different tool postures. First, an error index for tool tip deformation is introduced, and it considers the comprehensive deformation of the tool tip point instead of the translational deformation of the robot end-effector to offer a more accurate analysis of the machining error. Second, a rapid calculation method for cutter-workpiece engagement is developed, facilitating efficient calculation of cutting forces and enhancing the accuracy of deformation error calculation under various tool orientations. Finally, employing a particle swarm optimization algorithm with multiple constraints, including robot kinematics and tool interference, both tool orientation and robotic redundant angles are optimized to minimize tool error index at each cutter location. Through a comparison test using a simplified aeroengine casing, the proposed method demonstrates effective enhancement of the accuracy of robot milling processing compared with unoptimized and existing studies.</div></div>","PeriodicalId":21452,"journal":{"name":"Robotics and Computer-integrated Manufacturing","volume":"93 ","pages":"Article 102904"},"PeriodicalIF":9.1,"publicationDate":"2024-11-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142702827","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}