AI said, She said - How Users Perceive Consumer Scoring in Practice

As digitization continues, consumers are increasingly exposed to AI scoring decisions. However, currently lacking is a thorough understanding of how users’ misjudgments of an AI-supported system lead to it being rejected. Therefore, investigations are needed into the appropriation of such socio-technical systems in practice and how users describe their experience with algorithm-based scoring. To address this issue, we evaluated 1,003 user reviews of an app on car insurance that calculates premiums based on the consumers’ individual driving behavior. We find evidence that users develop their own folk theories to explain the algorithms with the help of situation-related experiences and that insufficient explanations lead to power asymmetries between consumers, the system, and the company. In particular, as a result of the different needs of the stakeholders, we uncover a fundamental conflict between computational risk assessment and the perceived agency to influence the score.