Kim Bjerge , Henrik Karstoft , Hjalte M.R. Mann , Toke T. Høye
{"title":"用于昆虫及其花卉环境延时摄影监测的深度学习管道","authors":"Kim Bjerge , Henrik Karstoft , Hjalte M.R. Mann , Toke T. Høye","doi":"10.1016/j.ecoinf.2024.102861","DOIUrl":null,"url":null,"abstract":"<div><div>Arthropods, including insects, represent the most diverse group and contribute significantly to animal biomass. Automatic monitoring of insects and other arthropods enables quick and efficient observation and management of ecologically and economically important targets such as pollinators, natural enemies, disease vectors, and agricultural pests. The integration of cameras and computer vision facilitates innovative monitoring approaches for agriculture, ecology, entomology, evolution, and biodiversity. However, studying insects and their interactions with flowers and vegetation in natural environments remains challenging, even with automated camera monitoring.</div><div>This paper presents a comprehensive methodology to monitor abundance and diversity of arthropods in the wild and to quantify floral cover as a key resource. We apply the methods across more than 10 million images recorded over two years using 48 insect camera traps placed in three main habitat types. The cameras monitor arthropods, including insect visits, on a specific mix of <em>Sedum</em> plant species with white, yellow and red/pink colored of flowers. The proposed deep-learning pipeline estimates flower cover and detects and classifies arthropod taxa from time-lapse recordings. However, the flower cover serves only as an estimate to correlate insect activity with the flowering plants.Color and semantic segmentation with DeepLabv3 are combined to estimate the percent cover of flowers of different colors. Arthropod detection incorporates motion-informed enhanced images and object detection with You-Only-Look-Once (YOLO), followed by filtering stationary objects to minimize double counting of non-moving animals and erroneous background detections. This filtering approach has been demonstrated to significantly decrease the incidence of false positives, since arthropods, occur in less than 3% of the captured images.</div><div>The final step involves grouping arthropods into 19 taxonomic classes. Seven state-of-the-art models were trained and validated, achieving <span><math><mrow><mi>F</mi><mn>1</mn></mrow></math></span>-scores ranging from 0.81 to 0.89 in classification of arthropods. Among these, the final selected model, EfficientNetB4, achieved an 80% average precision on randomly selected samples when applied to the complete pipeline, which includes detection, filtering, and classification of arthropod images collected in 2021. As expected during the beginning and end of the season, reduced flower cover correlates with a noticeable drop in arthropod detections. The proposed method offers a cost-effective approach to monitoring diverse arthropod taxa and flower cover in natural environments using time-lapse camera recordings.</div></div>","PeriodicalId":51024,"journal":{"name":"Ecological Informatics","volume":"84 ","pages":"Article 102861"},"PeriodicalIF":5.8000,"publicationDate":"2024-10-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"A deep learning pipeline for time-lapse camera monitoring of insects and their floral environments\",\"authors\":\"Kim Bjerge , Henrik Karstoft , Hjalte M.R. Mann , Toke T. Høye\",\"doi\":\"10.1016/j.ecoinf.2024.102861\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><div>Arthropods, including insects, represent the most diverse group and contribute significantly to animal biomass. Automatic monitoring of insects and other arthropods enables quick and efficient observation and management of ecologically and economically important targets such as pollinators, natural enemies, disease vectors, and agricultural pests. The integration of cameras and computer vision facilitates innovative monitoring approaches for agriculture, ecology, entomology, evolution, and biodiversity. However, studying insects and their interactions with flowers and vegetation in natural environments remains challenging, even with automated camera monitoring.</div><div>This paper presents a comprehensive methodology to monitor abundance and diversity of arthropods in the wild and to quantify floral cover as a key resource. We apply the methods across more than 10 million images recorded over two years using 48 insect camera traps placed in three main habitat types. The cameras monitor arthropods, including insect visits, on a specific mix of <em>Sedum</em> plant species with white, yellow and red/pink colored of flowers. The proposed deep-learning pipeline estimates flower cover and detects and classifies arthropod taxa from time-lapse recordings. However, the flower cover serves only as an estimate to correlate insect activity with the flowering plants.Color and semantic segmentation with DeepLabv3 are combined to estimate the percent cover of flowers of different colors. Arthropod detection incorporates motion-informed enhanced images and object detection with You-Only-Look-Once (YOLO), followed by filtering stationary objects to minimize double counting of non-moving animals and erroneous background detections. This filtering approach has been demonstrated to significantly decrease the incidence of false positives, since arthropods, occur in less than 3% of the captured images.</div><div>The final step involves grouping arthropods into 19 taxonomic classes. Seven state-of-the-art models were trained and validated, achieving <span><math><mrow><mi>F</mi><mn>1</mn></mrow></math></span>-scores ranging from 0.81 to 0.89 in classification of arthropods. Among these, the final selected model, EfficientNetB4, achieved an 80% average precision on randomly selected samples when applied to the complete pipeline, which includes detection, filtering, and classification of arthropod images collected in 2021. As expected during the beginning and end of the season, reduced flower cover correlates with a noticeable drop in arthropod detections. The proposed method offers a cost-effective approach to monitoring diverse arthropod taxa and flower cover in natural environments using time-lapse camera recordings.</div></div>\",\"PeriodicalId\":51024,\"journal\":{\"name\":\"Ecological Informatics\",\"volume\":\"84 \",\"pages\":\"Article 102861\"},\"PeriodicalIF\":5.8000,\"publicationDate\":\"2024-10-30\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Ecological Informatics\",\"FirstCategoryId\":\"93\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S1574954124004035\",\"RegionNum\":2,\"RegionCategory\":\"环境科学与生态学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"ECOLOGY\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Ecological Informatics","FirstCategoryId":"93","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1574954124004035","RegionNum":2,"RegionCategory":"环境科学与生态学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ECOLOGY","Score":null,"Total":0}
A deep learning pipeline for time-lapse camera monitoring of insects and their floral environments
Arthropods, including insects, represent the most diverse group and contribute significantly to animal biomass. Automatic monitoring of insects and other arthropods enables quick and efficient observation and management of ecologically and economically important targets such as pollinators, natural enemies, disease vectors, and agricultural pests. The integration of cameras and computer vision facilitates innovative monitoring approaches for agriculture, ecology, entomology, evolution, and biodiversity. However, studying insects and their interactions with flowers and vegetation in natural environments remains challenging, even with automated camera monitoring.
This paper presents a comprehensive methodology to monitor abundance and diversity of arthropods in the wild and to quantify floral cover as a key resource. We apply the methods across more than 10 million images recorded over two years using 48 insect camera traps placed in three main habitat types. The cameras monitor arthropods, including insect visits, on a specific mix of Sedum plant species with white, yellow and red/pink colored of flowers. The proposed deep-learning pipeline estimates flower cover and detects and classifies arthropod taxa from time-lapse recordings. However, the flower cover serves only as an estimate to correlate insect activity with the flowering plants.Color and semantic segmentation with DeepLabv3 are combined to estimate the percent cover of flowers of different colors. Arthropod detection incorporates motion-informed enhanced images and object detection with You-Only-Look-Once (YOLO), followed by filtering stationary objects to minimize double counting of non-moving animals and erroneous background detections. This filtering approach has been demonstrated to significantly decrease the incidence of false positives, since arthropods, occur in less than 3% of the captured images.
The final step involves grouping arthropods into 19 taxonomic classes. Seven state-of-the-art models were trained and validated, achieving -scores ranging from 0.81 to 0.89 in classification of arthropods. Among these, the final selected model, EfficientNetB4, achieved an 80% average precision on randomly selected samples when applied to the complete pipeline, which includes detection, filtering, and classification of arthropod images collected in 2021. As expected during the beginning and end of the season, reduced flower cover correlates with a noticeable drop in arthropod detections. The proposed method offers a cost-effective approach to monitoring diverse arthropod taxa and flower cover in natural environments using time-lapse camera recordings.
期刊介绍:
The journal Ecological Informatics is devoted to the publication of high quality, peer-reviewed articles on all aspects of computational ecology, data science and biogeography. The scope of the journal takes into account the data-intensive nature of ecology, the growing capacity of information technology to access, harness and leverage complex data as well as the critical need for informing sustainable management in view of global environmental and climate change.
The nature of the journal is interdisciplinary at the crossover between ecology and informatics. It focuses on novel concepts and techniques for image- and genome-based monitoring and interpretation, sensor- and multimedia-based data acquisition, internet-based data archiving and sharing, data assimilation, modelling and prediction of ecological data.