Designing for human-centered AI—Lessons learned from a case study in the clinical domain

AI tools for supporting, or even fully automating, human decision-making have been proposed in a variety of domains, promising faster and better quality of decisions. However, for high-stakes and critical decisions, humans are still required in the decision-making process. Despite the need for human involvement, the research core centers mainly around the technical issues of AI, i.e. how to develop better performing machine learning (ML) models, setting aside the issue of designing, developing, and evaluating AI tools that are to be used in a human-AI context. This focus has led to a lack of experience and guidance of designing and developing AI tools that support their users in a decision-making context, keeping the human in the loop.

In this paper, we outline our work on designing, developing, and evaluating a transparent AI-based tool to be used by non-AI experts, namely healthcare professionals. The work carried out had two parallel tracks. One focused on testing and implementing a suitable ML technique for sepsis diagnostics based on real patient data and applying explainable AI (XAI) techniques on the results to better enable healthcare professionals to understand and trust the analysis results. The other track included an iterative design process for developing a user-centered, transparent, and trustworthy sepsis diagnostic tool, evaluating whether the generated XAI explanations were fit for purpose. We present the process applied for intertwining these tracks during a common multidisciplinary development process, providing guidance how to conduct a human-centered AI (HCAI) project. We discuss lessons learned, and outline future work for the development of HCAI tools to be used by non-AI experts.