{"title":"Test method for measuring station-keeping with unmanned marine vehicles using sonar or optical sensors","authors":"Asish Ghoshal, A. Parnandi, R. Murphy","doi":"10.1145/2393091.2393122","DOIUrl":"https://doi.org/10.1145/2393091.2393122","url":null,"abstract":"This paper proposes a test method for measuring the ability of a USV or ROV to fixate on an underwater object, i.e., station-keeping. Station-keeping is needed to permit an operator or domain expert to stay focused on an area in an image long enough to identify objects, such as submerged cars and debris, or a condition, such as scour eroding the underwater footing of a bridge. This problem is different from traditional robot control, as the point is not to measure the positions of the robot and sensor payload but rather how well the system maintains the position of the object in the image. The test method uses the Lucas-Kanade optical flow algorithm in OpenCV to track an inexpensive raised plywood and wire fiducial. The rotational, translational, and root mean square (RMS) error is measured over a 3 minute period as well as number of image frames in which the fiducial was not visible. The method was demonstrated using a DIDSON acoustic camera, but is generalizable to other types of sonars and underwater video cameras.","PeriodicalId":423218,"journal":{"name":"Workshop on Performance Metrics for Intelligent Systems","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2012-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121225904","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Validation of the dynamics of an humanoid robot in USARSim","authors":"S. V. Noort, A. Visser","doi":"10.1145/2393091.2393128","DOIUrl":"https://doi.org/10.1145/2393091.2393128","url":null,"abstract":"This paper describes a model to replicate the dynamics of a walking robot inside USARSim. USARSim is an existing 3D simulator based on the Unreal Engine, which provides facilities for good quality rendering, physics simulation, networking, a highly versatile scripting language and a powerful visual editor. To model the dynamics of a walking robot the balance of the robot in relation with the contact points of the body with the environment has to be calculated. To guarantee a fast frame rate several approximations in this calculation have to be tried, and the performance (both in dynamics and computational effort) is evaluated in a number of experiments. This extension is made and validated for the humanoid robot Nao. On this basis many other applications become possible. A validated simulation allows us to develop and to experiment with typical robotic tasks before they are tested on a real robot.","PeriodicalId":423218,"journal":{"name":"Workshop on Performance Metrics for Intelligent Systems","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2012-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128691501","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
J. D. Victoria, O. Stryk, N. Policella, M. Gallant, A. Donati, Yang Gao
{"title":"Metrics for planetary rover planning & scheduling algorithms","authors":"J. D. Victoria, O. Stryk, N. Policella, M. Gallant, A. Donati, Yang Gao","doi":"10.1145/2393091.2393101","DOIUrl":"https://doi.org/10.1145/2393091.2393101","url":null,"abstract":"In addition to its utility in terrestrial-based applications, Automated Planning and Scheduling (P&S) has had a growing impact on space exploration. Such applications require an influx of new technologies to improve performance while not comprimising safety. As a result, a reliable method to rapidly assess the effectiveness of new P&S algorithms would be desirable to ensure the fulfillment of of all software requirements. This paper introduces RoBen, a mission-independent benchmarking tool that provides a standard framework for the evaluation and comparison of P&S algorithms. RoBen considers metrics derived from the model (the system on which the P&S algorithm will operate) as well as user input (e.g., desired problem complexity) to automatically generate relevant problems for quality assessment. A thorough description of the algorithms and metrics used in RoBen is provided, along with the preliminary test results of a P&S algorithm solving RoBen-generated problems.","PeriodicalId":423218,"journal":{"name":"Workshop on Performance Metrics for Intelligent Systems","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2012-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133839525","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A hybrid approach to 2D robotic map evaluation","authors":"Ross T. Creed, K. Georgiev, R. Lakaemper","doi":"10.1145/2393091.2393094","DOIUrl":"https://doi.org/10.1145/2393091.2393094","url":null,"abstract":"This article introduces the Temple Map Evaluation Toolkit (TMET), which is a tool for evaluating robotic maps produced by existing mapping algorithms. The toolkit performs ground truth based evaluation, i.e. it compares similarities between a map defined as ground truth and a target map. TMET allows for hybrid evaluation, since methods for pose based as well as grid based evaluation are implemented. For pose based evaluation, the user can define regions on the ground truth map which are handled as transformable sub-maps. TMET allows for evaluation of grid based maps as well as segment based maps, and therefore covers most of the representations of maps for existing mapping algorithms. The paper introduces the toolkit and the underlying design principles and algorithms. Experiments with maps from simulated as well as real world data are presented, demonstrating that the tool can be used to evaluate the quality of a map in a quantitative way.","PeriodicalId":423218,"journal":{"name":"Workshop on Performance Metrics for Intelligent Systems","volume":"38 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2012-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124430031","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Levels of human and robot collaboration for automotive manufacturing","authors":"Jane Shi, G. Jimmerson, Tom Pearson, R. Menassa","doi":"10.1145/2393091.2393111","DOIUrl":"https://doi.org/10.1145/2393091.2393111","url":null,"abstract":"United States Consortium for Automotive Research (USCAR) conducted a concept feasibility study in 2010--2011 to investigate critical requirements to implement fenceless (the long term goal) or minimally fenced (the short term goal) robotics work cells for automotive applications. One output of the study defines the levels of human and robot collaboration and addresses the levels of complexity that drive the probabilities of successful implementation. The development of these definitions was accomplished through interviews with technology providers, observation of current robot system installations, and discussions with automotive manufacturing engineers and robotic technical experts. In this paper, we attempt to categorize robotic systems for low, medium and high levels of human and robot collaboration with current state application examples in automotive body shop, automotive powertrain manufacturing and assembly, as well as in automotive general assembly. We propose potential human and robot collaboration applications in future state where sensors, when closely integrated with robotic systems with greater dynamic response and related new technology advancements, could enable a closer and more dynamic human and robot collaboration. Finally we highlight the assessment of the successful implementation probabilities for the low, medium, and high levels of human and robot collaborative applications.","PeriodicalId":423218,"journal":{"name":"Workshop on Performance Metrics for Intelligent Systems","volume":"41 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2012-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125665709","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Characterization of forward rectilinear-gait performance for a snake-inspired robot","authors":"James K. Hopkins, Satyandra K. Gupta","doi":"10.1145/2393091.2393119","DOIUrl":"https://doi.org/10.1145/2393091.2393119","url":null,"abstract":"Snake-inspired locomotion is much more maneuverable compared to conventional locomotion concepts and it enables a robot to navigate through rough terrain. A rectilinear gait is quite flexible and has the following benefits: functionality on a wide variety of terrains, enables a highly stable robot platform, and provides pure undulatory motion without passive wheels. However, historically speed has been a limitation for the locomotion type. In this paper, Fused Deposition Modeling (FDM) is utilized to reduced the weight and thereby increase the speed potential of a snake-inspired robot design based on a rectilinear gait. FDM also provides feasibility for development of complex and capable mechanism designs for executing rectilinear motion. The new design is analyzed, fabrication and evaluated based on various anchoring material velocity experiments.","PeriodicalId":423218,"journal":{"name":"Workshop on Performance Metrics for Intelligent Systems","volume":"24 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2012-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130563484","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Integrating occlusion monitoring into human tracking for robot speed and separation monitoring","authors":"W. Shackleford, R. Norcross, J. Marvel, S. Szabó","doi":"10.1145/2393091.2393124","DOIUrl":"https://doi.org/10.1145/2393091.2393124","url":null,"abstract":"Collaborative robots are used in close proximity to humans to perform a variety of tasks, while more traditional industrial robots are required to be stopped whenever a human enters their work-volumes. Instead of relying on physical barriers or merely detecting when someone enters the area, the collaborative system must monitor the position of every person who enters the work space in time for the robot to react. The TC 184/SC 2/WG 3 Industrial Safety group within the International Organization for Standard(ISO) is developing the standards to help ensure collaborative robots operate safely. Collaborative robots require sophisticated sensing technologies that must handle dynamic interactions between the robot and the human. One potential safety risk is the occlusion of a safety sensor's field of view due to placement of objects or the movement of people in front of a safety sensor. In this situation the robot could shut down as soon as even a single sensor was partially occluded. Unfortunately this could greatly diminish the extent to which the robot could work collaboratively. In this paper we examine how a human tracking system using multiple laser line scanners [3]was adapted to work with a robot Speed and Separation Monitoring (SSM) safety system and further modified to include occlusion monitoring.","PeriodicalId":423218,"journal":{"name":"Workshop on Performance Metrics for Intelligent Systems","volume":"83 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2012-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130740958","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A. Jacoff, Hui-Min Huang, Ann Virts, Anthony Downs, R. Sheh
{"title":"Emergency response robot evaluation exercise","authors":"A. Jacoff, Hui-Min Huang, Ann Virts, Anthony Downs, R. Sheh","doi":"10.1145/2393091.2393120","DOIUrl":"https://doi.org/10.1145/2393091.2393120","url":null,"abstract":"More than 60 robot test methods are being developed by a team led by the National Institute of Standards and Technology (NIST) with the sponsorship of U.S. Department of Homeland Security (DHS). These test methods are being specified and standardized under the standards development organization ASTM International. These standards are developed for the purposes of identifying the capabilities of mobile robots to help emergency response organizations assess the applicability of the robots.\u0000 The test methods are developed using an iterative process during which they are prototyped and validated by the participating researchers, developers, emergency response users, and robot manufacturers. We have conducted a series of evaluation exercises based on the test method implementations. These events were participated by representatives from all the different segments of the community. As such, these events present a unique opportunity for advancing the test methods, collecting capability data, and identifying robotic technology focusing issues. This paper describes an exercise event that this effort recently conducted.","PeriodicalId":423218,"journal":{"name":"Workshop on Performance Metrics for Intelligent Systems","volume":"80 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2012-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116701221","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"On the performance evaluation of a vision-based human-robot interaction framework","authors":"Junaed Sattar, G. Dudek","doi":"10.1145/2393091.2393096","DOIUrl":"https://doi.org/10.1145/2393091.2393096","url":null,"abstract":"This paper describes the performance evaluation of a machine vision-based human-robot interaction framework, particularly those involving human-interface studies. We describe a visual programming language called RoboChat, and a complimentary dialog engine which evaluates the need for confirmation based on utility and risk. Together, RoboChat and the dialog mechanism enable a human operator to send a series of complex instructions to a robot, with the assurance of confirmations in case of high task-cost or command uncertainty, or both. We have performed extensive human-interface studies to evaluate the usability of this framework, both in controlled laboratory conditions and in a variety of outdoors environments. One specific goal for the RoboChat scheme was to aid a scuba diver to operate and program an underwater robot in a variety of deployment scenarios, and the real-world validations were thus performed on-board the Aqua amphibious robot [4], in both underwater and terrestrial environments. The paper describes the details of the visual human-robot interaction framework, with an emphasis on the RoboChat language and the confirmation system, and presents a summary of the set of performance evaluation experiments performed both on- and off-board the Aqua vehicle.","PeriodicalId":423218,"journal":{"name":"Workshop on Performance Metrics for Intelligent Systems","volume":"23 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2012-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114313418","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Performance evaluation of robotic knowledge representation (PERK)","authors":"C. Schlenoff, S. Foufou, S. Balakirsky","doi":"10.1145/2393091.2393093","DOIUrl":"https://doi.org/10.1145/2393091.2393093","url":null,"abstract":"In this paper, we explore some ways in which symbolic knowledge representations have been evaluated in the past and provide some thoughts on what should be considered when applying and evaluating these types of knowledge representations for real-time robotics applications. The emphasis of this paper is that the robotic applications require real-time access to information, which has not been one of the aspects measured in traditional symbolic representation evaluation approaches.","PeriodicalId":423218,"journal":{"name":"Workshop on Performance Metrics for Intelligent Systems","volume":"24 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2012-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114725201","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}