Unveiling Explainable AI in Healthcare: Current Trends, Challenges, and Future Directions

This overview investigates the evolution and current landscape of eXplainable Artificial Intelligence (XAI) in healthcare, highlighting its implications for researchers, technology developers, and policymakers. Following the PRISMA protocol, we analyzed 89 publications from January 2000 to June 2024, spanning 19 medical domains, with a focus on Neurology and Cancer as the most studied areas. Various data types are reviewed, including tabular data, medical imaging, and clinical text, offering a comprehensive perspective on XAI applications. Key findings identify significant gaps, such as the limited availability of public datasets, suboptimal data preprocessing techniques, insufficient feature selection and engineering, and the limited utilization of multiple XAI methods. Additionally, the lack of standardized XAI evaluation metrics and practical obstacles in integrating XAI systems into clinical workflows are emphasized. We provide actionable recommendations, including the design of explainability‐centric models, the application of diverse and multiple XAI methods, and the fostering of interdisciplinary collaboration. These strategies aim to guide researchers in building robust AI models, assist technology developers in creating intuitive and user‐friendly AI tools, and inform policymakers in establishing effective regulations. Addressing these gaps will promote the development of transparent, reliable, and user‐centred AI systems in healthcare, ultimately improving decision‐making and patient outcomes.