Developing an Accounting Virtual Assistant Through Supervised Fine-Tuning (SFT) of a Small Language Model (SLM)

Abstract

The development of an in-house accounting bot—an artificial intelligence (AI) assistant capable of generating internally structured bookkeeping double-entry posting schemes—is explored in this paper. The processes of curating a suitable dataset, selecting, and fine-tuning a seven-billion-parameter language model, categorized as a small language model (SLM) (SLMs typically refer to models with fewer than 10 billion parameters, whereas medium-sized models often have 14B parameters, and large-scale models exceed 70B), are described. A human-evaluated benchmark is also presented to assess model performance. To achieve efficient supervised fine-tuning (SFT), low-rank adaptation (LoRA) was employed, significantly reducing memory requirements by using a small set of trainable parameters while maintaining model expressiveness. The process of backpropagation was further optimized using Unsloth, a high-performance training framework designed for efficient video memory usage and flash attention mechanisms, which accelerates adaptation and reduces memory overhead. The model whose layers were updated is called QwenCoder2.5. It was selected with the presumption that it would be able to learn how to generate and examine bookkeeping patterns generated by accounting information system (AIS) over a 17-year history. This proof of concept aims to support researchers and practitioners exploring the integration of generative AI in accounting by providing insights into both the benefits and challenges of AI-driven automation in bookkeeping tasks. The study demonstrates how an SLM can be fine-tuned on a proprietary dataset of journal posting schemes to assist accountants, auditors, and financial analysts while also facilitating synthetic data generation. Challenges related to AI, data preprocessing, fine-tuning optimization, and evaluation methodology are introduced and examined.