Reassessing the predictive power of bedfinder: insights into machine learning for subglacial bedform detection – Comments on ‘Automatic identification of streamlined subglacial bedforms using machine learning: an open-source Python approach’

The integration of machine learning (ML) into geomorphological research presents significant opportunities for automating the identification of streamlined subglacial bedforms. In their study, Abrahams et al. (2024) introduce bedfinder, an open-source Python tool designed to detect subglacial features with high efficiency. While the tool demonstrates promise, this discussion highlights critical challenges and areas for refinement. The representativeness of the training data set, dominated by sedimentary bed conditions, limits the model's generalizability to regions with diverse bedrock compositions. Additionally, the reliance on binary classifications oversimplifies complex geomorphic settings, reducing the model's adaptability. Performance metrics such as F1 scores, though favourable, warrant cautious interpretation due to class imbalances that may skew predictions. Furthermore, the integration of filtering techniques, while enhancing precision, raises concerns about potential biases from manual data curation. To enhance scientific rigour, future efforts should incorporate diverse data sets, conduct comprehensive evaluations of filtering methods, and explore modular approaches for greater applicability. Addressing these challenges will not only strengthen bedfinder but also contribute to the evolving role of ML in advancing glacial and geomorphological research. This contribution provides a constructive critique to guide future improvements and interdisciplinary applications of this innovative tool.

Abrahams et al. (2024) present a cutting-edge approach to the identification of streamlined subglacial bedforms, which are crucial in understanding past glacial dynamics and their impact on geomorphology. The authors have developed an open-source Python tool, bedfinder, that employs supervised ML algorithms – Random Forest and XGBoost – to automate the identification of these features across deglaciated landscapes. This tool is an ambitious attempt to enhance efficiency and accuracy in a domain traditionally burdened by labour-intensive, subjective, and time-consuming manual processes. The authors also provide a thorough validation of the model using an extensive data set of known subglacial bedforms from various regions in the Northern Hemisphere.

While the study contributes valuable insights into how ML can automate the mapping of complex glacial features, several critical issues arise from the selection of data, the application of ML methods, and the interpretation of results. Here, we will highlight some of these issues and offer suggestions to enhance the scientific robustness of the study and its potential applications. The concerns outlined here include the representativeness of the training data, the challenges in interpreting the performance metrics, and the potential over-simplifications in model design and validation.

A central issue in Abrahams et al. (2024) lies in the selection of the training data. The authors utilize a data set derived primarily from sedimentary bed conditions, which introduces inherent biases in the model's predictive capabilities. Subglacial bedforms, such as streamlined ridges and grooves, are deeply influenced by a variety of factors, including the underlying bedrock composition, topography, and glacial dynamics. As such, the diversity of the landscape in which these features occur plays a crucial role in determining the model's ability to generalize across different glacial environments. The data set employed in this study, however, is heavily weighted towards sedimentary bed, with only limited representation of crystalline bed conditions. This imbalance limits the model's applicability to regions that predominantly feature crystalline or mixed bedrock, thereby potentially reducing its accuracy when applied to environments where these conditions prevail. As a result, the model may struggle to accurately detect and classify subglacial bedforms in these underrepresented areas, which could lead to misinterpretations of the underlying geomorphological processes.

The sedimentary dominance in the training data implies that the model is optimized for environments where glacial erosion is more pronounced and where the bedforms are likely to exhibit greater contrast in relief. However, such conditions may not accurately reflect the range of bedform characteristics found in other glacial terrains. For example, glacial features formed over crystalline bedrock are often more subdued in terms of their topographic expression, and they may exhibit unique forms not adequately represented by the existing data set (Skyttä et al. 2023; Courtney-Davies et al. 2024). As a result, the model may struggle to detect such features with high accuracy, leading to underreporting or misclassification of subglacial bedforms in non-sedimentary terrains.

Furthermore, the simplified binary classification of bed conditions into sedimentary vs. crystalline, and topography into constrained vs. unconstrained categories, oversimplifies the inherent complexity of glacial landscapes. Real-world geomorphological systems often exhibit a continuum of bedrock and topographic types, with gradual transitions between these categories (Tani 2013; Stewart & Jamieson 2018). The choice to categorize bed conditions so rigidly ignores the possibility of more complex hybrid environments, such as regions where bedrock characteristics and topographic constraints exist in subtle or transitional forms. This approach thus limits the flexibility of the model to handle areas that do not fit neatly into these predefined categories.

In fact, glacial bedforms in areas with mixed or transitional bedrock may behave differently under the same climatic conditions (Oetting et al. 2022; Olesen et al. 2023). For instance, areas where sedimentary and crystalline bedrocks coexist may exhibit interactions between ice flow, erosion, and deposition that do not fit the typical patterns observed in homogeneous environments (Xie et al. 2022; Huang et al. 2023). Such regions could present challenges for the model, as the unique interplay between geological and topographical factors would not be captured by the limited training data set.

The solution to this issue lies in expanding the training data set to better represent the diversity of glacial environments. Future iterations of the model should incorporate data sets from regions that exhibit a wider variety of bedrock types (including volcanic, metamorphic and mixed compositions), as well as topographic features that span the entire spectrum from constrained to unconstrained settings. By integrating data from areas such as the Arctic, Antarctic, and other high-latitude glaciated regions, the model would gain a broader understanding of how subglacial bedforms manifest under different conditions. Additionally, the classification system could be refined to capture the nuanced variations that exist within different bedrock and topographic types. Instead of relying on a binary classification, the authors could develop a multi-tiered system that includes subcategories for mixed or transitional environments. This would allow for more precise model predictions in regions where the interaction of various geological and topographical factors is crucial. Moreover, incorporating regional-specific metadata and environmental variables, such as glacial flow dynamics, sediment transport characteristics, and variations in ice-sheet thickness, could enhance the predictive power of the model. These variables would provide the necessary context for interpreting the specific glacial history and bedform characteristics of different regions, thus enabling the model to adjust its predictions based on the unique environmental conditions of each location.

Abrahams et al. (2024) report impressive performance metrics, with F1 scores exceeding 94% for both Random Forest and the ensemble models. However, these figures must be viewed in light of the significant class imbalance present in the training data. The data set contains over 600 000 positive-relief features, but only a small proportion of these are identified as true subglacial bedforms. This creates an inherent imbalance between the classes of interest (glacial bedforms) and non-glacial features (e.g. rock outcrops, sedimentary anomalies, etc.), which has the potential to skew model performance.

In ML tasks involving class imbalance, standard evaluation metrics like accuracy can be misleading, as models can achieve high accuracy simply by predicting the dominant class (in this case, ‘non-bedform’ features) without ever identifying true bedforms. To account for this, the authors use additional metrics, such as precision, recall, and the F1 score, which better reflect the model's ability to detect true positives while minimizing false positives and false negatives. While these metrics indicate a strong model performance, the authors do not sufficiently address the influence of the class imbalance on these results.

The authors correctly point out that the model's tendency to overpredict false positives (especially in OOD regions) is a result of prioritizing the detection of bedforms rather than avoiding false positives. While this approach may be acceptable for applications where missing a bedform is more problematic than misclassifying a non-bedform, it remains essential to fully understand the implications of such decisions. False positives – where non-glacial features are misclassified as bedforms – could significantly alter the interpretation of a landscape, leading to incorrect assumptions about past ice-flow dynamics and glacial behaviour.

Furthermore, the false negative rate, while implicitly addressed through the model's emphasis on recovering more bedforms, is not sufficiently explored. A higher false positive rate may be preferable in certain circumstances, but the trade-off with false negatives needs careful evaluation. For example, in a region with complex geomorphology, the presence of non-glacial features might lead to substantial false positive detections, which could undermine the scientific value of the model's predictions.

To improve the robustness of the results, it would be beneficial to conduct a more comprehensive evaluation of the trade-offs between false positives and false negatives. The authors could employ a precision-recall curve (Saito & Rehmsmeier 2017; Fu et al. 2019; Williams 2021) to assess the model's performance in greater detail. This would allow for a better understanding of how different threshold values for classification impact the balance between precision and recall, ultimately providing a clearer picture of the model's practical utility in various contexts. In addition, applying cross-validation techniques using different training and test sets would provide a more rigorous assessment of the model's generalizability. Since the data set is likely to contain spatial and temporal dependencies, the authors could consider employing spatial cross-validation, where training and test sets are split based on geographical regions rather than random selection (Allen & Kim 2020; Palm et al. 2023). This approach would more accurately capture the model’s ability to generalize across different spatial scales and environmental conditions, further validating its effectiveness for real-world applications.

The authors introduce a scientifically driven filtering method, referred to as McKenzie et al. (2022) filtering, which aims to narrow the data set to only those features that are likely to be streamlined subglacial bedforms. This method is intended to mitigate the errors associated with false positives and non-glacial features, ensuring that the model is trained on a high-quality subset of the data. However, while filtering improves model performance by excluding spurious features, it introduces a level of subjectivity into the process that warrants further scrutiny.

The reliance on manual filtering – albeit guided by scientific principles – can introduce biases into the data set that are difficult to quantify. For example, the decision to exclude certain bedform types or morphologies based on subjective criteria may unintentionally eliminate features that are of scientific interest or that challenge conventional definitions of subglacial bedforms. As a result, the model may be trained on a data set that is more homogeneous than the actual diversity of features encountered in the field.

Moreover, the combination of filtering methods – such as McKenzie filtering alongside Near Miss – deserves more attention in terms of its effectiveness. The authors suggest that the integration of these methods results in improved detection accuracy, but the rationale for this combination is not fully explained. Is it the scientific knowledge embedded in McKenzie filtering that enhances model performance, or is it the statistical adjustment provided by Near Miss? The authors do not sufficiently discuss how these methods interact, which makes it difficult to assess the true impact of their combined use.

To enhance the scientific rigour of the filtering process, the authors could benefit from conducting a more transparent comparison of different filtering strategies. Ablation studies (Cosmo et al. 2024; Tseng et al. 2024) could be employed to evaluate the independent contribution of each filtering method and determine whether the combination of McKenzie filtering and Near Miss is truly optimal. Additionally, the authors could test the model's performance on unfiltered data sets to assess whether the added complexity of filtering results in a tangible improvement in predictive accuracy or merely serves to reduce the diversity of features considered. In the future, incorporating a flexible, data-driven filtering mechanism – perhaps informed by unsupervised clustering or density-based spatial analysis – could help automate the feature selection process while reducing potential biases introduced by manual choices. This would allow the model to adjust to new data without overfitting to pre-existing assumptions.

Abrahams et al. (2024) present a significant advance in automating the identification of streamlined subglacial bedforms, a task traditionally dominated by manual methods that are prone to errors and biases. While the study demonstrates the potential of machine learning to transform geomorphology, several important challenges remain. These include issues related to data selection, model performance interpretation, and the integration of scientific filtering methods. By addressing these concerns, the authors could significantly improve the robustness and generalizability of the tool. Future iterations of bedfinder could benefit from expanding the training data set, refining the model's ability to handle class imbalance, and introducing a more flexible filtering mechanism.

With these improvements, bedfinder could become a cornerstone for future research in glacial geomorphology, offering a powerful, reproducible and accessible tool for understanding past ice dynamics and advancing the field of glaciology.

ML: writing – original draft. HZ: formal analysis. TY: conceptualization, supervision, funding acquisition, writing – review and editing.