Towards unveiling sensitive and decisive patterns in explainable AI with a case study in geometric deep learning

Abstract

The interpretability of machine learning models has gained increasing attention, particularly in scientific domains where high precision and accountability are crucial. This research focuses on distinguishing between two critical data patterns—sensitive patterns (model related) and decisive patterns (task related)—which are commonly used as model interpretations but often lead to confusion. Specifically, this study compares the effectiveness of two main streams of interpretation methods: post-hoc methods and self-interpretable methods, in detecting these patterns. Recently, geometric deep learning (GDL) has shown superior predictive performance in various scientific applications, creating an urgent need for principled interpretation methods. Here, therefore, we conduct our study using several representative GDL applications as case studies. We evaluate 13 interpretation methods applied to 3 major GDL backbone models, using 4 scientific datasets to assess how well these methods identify sensitive and decisive patterns. Our findings indicate that post-hoc methods tend to provide interpretations better aligned with sensitive patterns, whereas certain self-interpretable methods exhibit strong and stable performance in detecting decisive patterns. Moreover, our study offers valuable insights into improving the reliability of these interpretation methods. For example, ensembling post-hoc interpretations from multiple models trained on the same task can effectively uncover the task’s decisive patterns. Interpreting decisions made by machine learning systems remains difficult. Here Zhu et al. test interpretability methods on their ability to identify model-related and task-related patterns.