Should AI-enabled medical devices be explainable?

Despite its exponential growth, artificial intelligence (AI) in healthcare faces various challenges. One of them is a lack of explainability of AI medical devices, which arguably leads to insufficient trust in AI technologies, quality, and accountability and liability issues. The aim of this paper is to examine whether, why and to what extent AI explainability should be demanded with relation to AI-enabled medical devices and their outputs. Relying on a critical analysis of interdisciplinary literature on this topic and an empirical study, we conclude that the role of AI explainability in the medical AI context is a limited one. If narrowly defined, AI explainability principle is capable of addressing only a limited range of challenges associated with AI and is likely to reach fewer goals than sometimes expected. The study shows that, instead of technical explainability of medical AI devices, most stakeholders need more transparency around its development and quality assurance process.