Evaluating Explainable AI with Humans: Trust, Reliance, Understanding, and Performance

Abstract

Explainable AI (XAI) is often evaluated through claims about trust, reliance, understanding, and performance, yet studies measure these constructs differently and frequently overlook confounding factors. This PRISMA-guided systematic review with narrative synthesis examines human-centered XAI evaluation across these four constructs. The review covers publications from 2015 to 2025. A merged search combined an OpenAlex-informed DOI/title catalog, prior surveys, citation chasing, and a Scopus coverage-completion stage identifying 9,026 records. A targeted revision update added two eligible publications, yielding 119 validated full-text publications. The synthesis highlights three patterns: trust and reliance can diverge; perceived understanding can exceed objective understanding; and interface salience or perceived accuracy can confound explanation effects. Aggregate advice acceptance, for example, cannot distinguish beneficial reliance from automation bias. We therefore recommend measuring acceptance separately for correct and incorrect AI outputs, testing understanding beyond self-report, documenting study context and interface conditions, and evaluating explanations under uncertainty or model failure. The review contributes construct definitions, a synthesis across designs and domains, an analysis of evaluation measures, and practical reporting guidelines. It also examines generated explanations and citation-based interfaces, supporting more reproducible and interpretable human-centered XAI evaluations.