{"title":"基于双目视觉的柑橘采摘点检测和定位","authors":"Chaojun Hou, Jialiang Xu, Yu Tang, Jiajun Zhuang, Zhiping Tan, Weilin Chen, Sheng Wei, Huasheng Huang, Mingwei Fang","doi":"10.1007/s11119-024-10169-2","DOIUrl":null,"url":null,"abstract":"<p>Accurate localization of picking points in non-structural environments is crucial for intelligent picking of ripe citrus with a harvesting robot. However, citrus pedicels are too small and resemble other background objects in color, making it challenging to detect and localize the picking point of citrus fruits. This work presents a novel approach for detecting and localizing citrus picking points using binocular vision. First, the convolutional block attention module (CBAM) attention model is integrated into the backbone network of Mask R-CNN to increase the feature extraction for citrus pedicels, and the soft-non maximum suppression (Soft-NMS) strategy is used in the region proposal network to enhance the detection performance of citrus pedicel. Second, to accurately associate the citrus fruit with the best detected pedicel, a maximum discrimination criterion is proposed by integrating the confidence score of the detected pedicel and the degree of positional connectivity between the pedicel and the fruit. Finally, to reduce matching errors and improve computational efficiency, a rapid and robust matching method based on the normalized cross-correlation was applied to search the picking point within the line segment between the left and right images. The experimental results show that the precision, recall and F1-score for pedicel detection are 95.04%, 88.11%, and 91.44%, respectively, which are improvement of 13.00%, 7.84%, and 10.30% compared to the original Mask R-CNN. The mean absolute error (MAE) for the localizing the citrus picking point is 8.63 mm and the mean relative error (MRE) is 2.76%. The MRE was significantly reduced by at least 1.2% compared to the stereo matching methods belief-propagation (BP), semi-global block matching (SGBM), and block matching (BM), respectively. This study provides an effective method for the precise detection and localization of citrus picking point for a harvesting robot.</p>","PeriodicalId":20423,"journal":{"name":"Precision Agriculture","volume":null,"pages":null},"PeriodicalIF":5.4000,"publicationDate":"2024-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Detection and localization of citrus picking points based on binocular vision\",\"authors\":\"Chaojun Hou, Jialiang Xu, Yu Tang, Jiajun Zhuang, Zhiping Tan, Weilin Chen, Sheng Wei, Huasheng Huang, Mingwei Fang\",\"doi\":\"10.1007/s11119-024-10169-2\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p>Accurate localization of picking points in non-structural environments is crucial for intelligent picking of ripe citrus with a harvesting robot. However, citrus pedicels are too small and resemble other background objects in color, making it challenging to detect and localize the picking point of citrus fruits. This work presents a novel approach for detecting and localizing citrus picking points using binocular vision. First, the convolutional block attention module (CBAM) attention model is integrated into the backbone network of Mask R-CNN to increase the feature extraction for citrus pedicels, and the soft-non maximum suppression (Soft-NMS) strategy is used in the region proposal network to enhance the detection performance of citrus pedicel. Second, to accurately associate the citrus fruit with the best detected pedicel, a maximum discrimination criterion is proposed by integrating the confidence score of the detected pedicel and the degree of positional connectivity between the pedicel and the fruit. Finally, to reduce matching errors and improve computational efficiency, a rapid and robust matching method based on the normalized cross-correlation was applied to search the picking point within the line segment between the left and right images. The experimental results show that the precision, recall and F1-score for pedicel detection are 95.04%, 88.11%, and 91.44%, respectively, which are improvement of 13.00%, 7.84%, and 10.30% compared to the original Mask R-CNN. The mean absolute error (MAE) for the localizing the citrus picking point is 8.63 mm and the mean relative error (MRE) is 2.76%. The MRE was significantly reduced by at least 1.2% compared to the stereo matching methods belief-propagation (BP), semi-global block matching (SGBM), and block matching (BM), respectively. This study provides an effective method for the precise detection and localization of citrus picking point for a harvesting robot.</p>\",\"PeriodicalId\":20423,\"journal\":{\"name\":\"Precision Agriculture\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":5.4000,\"publicationDate\":\"2024-07-28\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Precision Agriculture\",\"FirstCategoryId\":\"97\",\"ListUrlMain\":\"https://doi.org/10.1007/s11119-024-10169-2\",\"RegionNum\":2,\"RegionCategory\":\"农林科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"AGRICULTURE, MULTIDISCIPLINARY\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Precision Agriculture","FirstCategoryId":"97","ListUrlMain":"https://doi.org/10.1007/s11119-024-10169-2","RegionNum":2,"RegionCategory":"农林科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"AGRICULTURE, MULTIDISCIPLINARY","Score":null,"Total":0}
Detection and localization of citrus picking points based on binocular vision
Accurate localization of picking points in non-structural environments is crucial for intelligent picking of ripe citrus with a harvesting robot. However, citrus pedicels are too small and resemble other background objects in color, making it challenging to detect and localize the picking point of citrus fruits. This work presents a novel approach for detecting and localizing citrus picking points using binocular vision. First, the convolutional block attention module (CBAM) attention model is integrated into the backbone network of Mask R-CNN to increase the feature extraction for citrus pedicels, and the soft-non maximum suppression (Soft-NMS) strategy is used in the region proposal network to enhance the detection performance of citrus pedicel. Second, to accurately associate the citrus fruit with the best detected pedicel, a maximum discrimination criterion is proposed by integrating the confidence score of the detected pedicel and the degree of positional connectivity between the pedicel and the fruit. Finally, to reduce matching errors and improve computational efficiency, a rapid and robust matching method based on the normalized cross-correlation was applied to search the picking point within the line segment between the left and right images. The experimental results show that the precision, recall and F1-score for pedicel detection are 95.04%, 88.11%, and 91.44%, respectively, which are improvement of 13.00%, 7.84%, and 10.30% compared to the original Mask R-CNN. The mean absolute error (MAE) for the localizing the citrus picking point is 8.63 mm and the mean relative error (MRE) is 2.76%. The MRE was significantly reduced by at least 1.2% compared to the stereo matching methods belief-propagation (BP), semi-global block matching (SGBM), and block matching (BM), respectively. This study provides an effective method for the precise detection and localization of citrus picking point for a harvesting robot.
期刊介绍:
Precision Agriculture promotes the most innovative results coming from the research in the field of precision agriculture. It provides an effective forum for disseminating original and fundamental research and experience in the rapidly advancing area of precision farming.
There are many topics in the field of precision agriculture; therefore, the topics that are addressed include, but are not limited to:
Natural Resources Variability: Soil and landscape variability, digital elevation models, soil mapping, geostatistics, geographic information systems, microclimate, weather forecasting, remote sensing, management units, scale, etc.
Managing Variability: Sampling techniques, site-specific nutrient and crop protection chemical recommendation, crop quality, tillage, seed density, seed variety, yield mapping, remote sensing, record keeping systems, data interpretation and use, crops (corn, wheat, sugar beets, potatoes, peanut, cotton, vegetables, etc.), management scale, etc.
Engineering Technology: Computers, positioning systems, DGPS, machinery, tillage, planting, nutrient and crop protection implements, manure, irrigation, fertigation, yield monitor and mapping, soil physical and chemical characteristic sensors, weed/pest mapping, etc.
Profitability: MEY, net returns, BMPs, optimum recommendations, crop quality, technology cost, sustainability, social impacts, marketing, cooperatives, farm scale, crop type, etc.
Environment: Nutrient, crop protection chemicals, sediments, leaching, runoff, practices, field, watershed, on/off farm, artificial drainage, ground water, surface water, etc.
Technology Transfer: Skill needs, education, training, outreach, methods, surveys, agri-business, producers, distance education, Internet, simulations models, decision support systems, expert systems, on-farm experimentation, partnerships, quality of rural life, etc.