Black Box Analytics and Ethical Decision Making

Abstract

Using an experiment with participants having management experience, we examine sales target setting decisions using an analytics-based forecasting system in a situation involving an ethical dilemma. Specifically, participants have private information that the forecast significantly underestimates likely sales, making the suggested target easily achievable. We explore the extent to which participants act unethically by not adjusting the sales target upwards. We employ a 2x2 between-subjects design, manipulating forecasting system transparency (opaque vs transparent) and accountability both as a measured continuous variable and with the use of a prompt either before (pre-prompt) or after (post-prompt) the adjustment decision. We find that participants make less ethical decisions when the system is opaque and more ethical decisions when they feel greater accountability. The effect of accountability is greatest when the system is opaque. We also examine reasons provided for less ethical decisions and find that the least ethical participants use more rationalizations than those whose decisions are not as unethical. Our results suggest that organizations should endeavor to make data analytics systems transparent to decision making users. However, when they cannot, they should ensure that decision makers feel accountable for their decisions; for example, with a prompt or decision aid.