The role of user feedback in enhancing understanding and trust in counterfactual explanations for explainable AI

Abstract

Counterfactual explanations (CEs) have emerged as a viable solution for generating comprehensible explanations in the context of explainable artificial intelligence (XAI). A CE provides actionable information to users on how to achieve the desired outcome from a machine learning (ML) model with minimal modifications to the input. XAI is crucial for improving transparency and reliability in AI systems, especially for meeting regulations like the General Data Protection Regulation (GDPR) or the European AI Act. However, the integration of CEs into XAI frameworks and their effectiveness in enhancing user trust and cognitive learning remains uncertain and requires further research. We have developed a user study to face this challenge with two user input-driven counterfactual generation XAI approaches: (i) User Feedback-based Counterfactual Explanation (UFCE) and (ii) Diverse Counterfactual Explanation (DiCE). They are integrated within a game-inspired online platform that enables direct comparisons between them. We compared the task performance, understanding, satisfaction, and trust between control and experimental groups, with a total of 101 participants. After curating the collected data, we had 70 users (24 in the control group) who successfully completed the experiment. Participants in the experimental group received explanations generated by UFCE or DiCE. Findings show that explanations generated by UFCE improve users’ learning experiences, resulting in better task performance, comprehension, satisfaction, and trust. Moreover, participants who interacted with UFCE exhibited significantly higher reliance on suggestions than those who interacted with DiCE, what was supported by statistical validation. These results highlight the significance of human-centered XAI methods and promote meaningful cognitive engagement for users. Furthermore, the game-inspired platform is implemented as open-source to promote Open Science, and it is made publicly available along with data collected in the user study to support further investigations and to ensure reproducibility of reported results.