Can ChatGPT be a knowledgeable and accurate ethical accountant? Assessing the overall usefulness of ChatGPT’s answers to ethical dilemmas

Abstract

This paper aims to assess the overall usefulness of ChatGPT regarding ethical dilemmas in accounting by exploring the tool’s ability to produce accurate answers, as well as reliably and properly identify the relevant legal frameworks in its justification. This study employs a quasi-experimental method and an exploratory perspective to evaluate ChatGPT 4.0′s responses to questions from the Portuguese Order of Certified Accountants exams. Consistency and robustness tests were conducted by varying and repeating the prompts, and the outputs (ChatGPT responses) were assessed through a content analysis method by comparing them with official answer keys (accuracy) and their justification (legal frameworks). The findings indicate that, throughout the process, ChatGPT’s performance is influenced by prompt design and repetition, and it consistently lacks accuracy, namely when professional judgment is required. Besides, although it is generally capable of recognising the relevant legal frameworks, its justifications were usually vague. Moreover, whenever more precise answers were provided, evidence of hallucinations was commonly found. Therefore, while the responses are often persuasive and structured, their reliability is limited, raising concerns about the model’s epistemic soundness and reducing its overall usefulness, which highlights the need for users to exercise caution when relying on their outputs. To the best of the authors’ knowledge, this is the first research to analyse in depth the characteristics within ChatGPT responses, specifically regarding professional ethics in an accounting proficiency exam. This study offers empirical evidence on ChatGPT’s limitations and potential as a supplementary tool in accounting ethics. It contributes to ongoing debates about the role of generative AI in professional education and practice, providing insights for educators and practitioners regarding its responsible and cautious integration. While promising as a discussion aid, ChatGPT still cannot replace the critical thinking and contextual reasoning required for ethical decision-making.