Challenges in the evaluation of earth observation products: Accuracy assessment case study using convolutional neural networks

Abstract

Earth observation is essential for monitoring natural resources and the Earth’s climate. However, the accuracy of earth observation products is sometimes not adequately evaluated, mainly when deep learning (DL) is used. One reason is that the DP and remote sensing communities have different ways of evaluating accuracy. The DL community tends to use single metrics to summarize results, which the remote sensing community overlooks. On the other hand, the remote sensing community emphasizes transparency in map creation, documenting sampling methods, and error matrices, which is not a priority for the DL community. Therefore, a significant challenge in applying DP methods for earth observation is the lack of evaluation using the established metrics of the remote sensing community, which are not commonly used in the DL community. In addition, assessing the accuracy of deep learning models adds another layer of complexity to the process. To tackle this challenge, we conducted a case study on creating a map of settlements in Bavaria using CNNs and satellite images. We then evaluated the resulting map according to the recommendations found in remote sensing literature. Our evaluation revealed that the CNNs we trained had an Overall Accuracy of over 97%. Since the remote sensing literature recommends not reporting only Overall Accuracy as a metric, especially for class imbalances, we also specified the confusion matrices with sample and proportion counts and the estimators for a 95% confidence interval as additional metrics and performed a visual evaluation. Our evaluation is based on our previous examination of recommended assessment practices. Our aim in presenting this overview, along with the case study, is to help readers identify potential issues and offer guidance for evaluating their earth observation products.