User experience key performance indicators for industrial IoT systems: A multivocal literature review

In software systems, user experience (UX) is a quality that provides value to its intended users. Therefore, we consider it critical to manage UX and its link to business value quantitatively. Key performance indicators (KPIs) have been widely adopted for objectively measuring factors that influence business success, including measuring UX characteristics and their business contribution in the context of software systems in general. Yet, measuring UX in the context of complex Internet of Things (IoT) systems, especially Industrial IoT (IIoT) systems, remains a challenge.

This paper presents a multivocal literature review (MLR) of the UX-KPIs proposed in the context of (I)IoT systems, along with factors influencing UX and challenges of quantifying UX. We investigate the status quo and derive recommendations for measuring the UX of (I)IoT systems.

Our analysis of 64 publications shows that UX has so far received relatively little attention in the context of (I)IoT. Although we identified 605 KPIs, 78 influencing factors, and 119 challenges, most of the UX features discussed were defined in a broader context of software systems, i.e., they were less specific to IoT, but rather focused on universal system quality characteristics such as functional suitability, performance efficiency, usability, and reliability.

Finally, no generally accepted approach for measuring UX in (I)IoT systems and how these contribute to the overall business value has been proposed yet. We also found little evidence that improving UX or achieving a good UX currently plays an explicit role in the design and development of IoT systems.