Explainable AI: definition and attributes of a good explanation for health AI

Abstract

Proposals of artificial intelligence (AI) solutions based on more complex and accurate predictive models are becoming ubiquitous across many disciplines. As the complexity of these models increases, there is a tendency for transparency and users’ understanding to decrease. This means accurate prediction alone is insufficient to make an AI-based solution truly useful. For the development of healthcare systems, this raises new issues for accountability and safety. How and why an AI system made a recommendation may necessitate complex explanations of the inner workings and reasoning processes. While research on explainable AI (XAI) has grown significantly in recent years, and the demand for XAI in medicine is high, determining what constitutes a good explanation is ad hoc and providing adequate explanations remains a challenge. To realise the potential of AI, it is critical to shed light on two fundamental questions of explanation for safety–critical AI such as health-AI that remain unanswered: (1) What is an explanation in health-AI? And (2) What are the attributes of a good explanation in health-AI? In this study and possibly for the first time we studied published literature, and expert opinions from a diverse group of professionals reported from a two-round Delphi study. The research outputs include (1) a proposed definition of explanation in health-AI, and (2) a comprehensive set of attributes that characterize a good explanation in health-AI.