A new fixed-point simulation methodology for on-device AI based on large language models

Large language models (LLMs) have demonstrated outstanding performance across various natural language processing tasks, and their utilization in on-device environments is gradually increasing. This paper proposes a full integer arithmetic (fixed-point arithmetic) methodology utilizing fixed-point simulation to optimize the LLaMA3-8B-Instruct model for on-device hardware development. The proposed approach enables integer computations without performance degradation on the MMLU benchmark. Conventional quantization methods primarily focus on integer conversion of weight matrix multiplication operations; however, they require subsequent floating-point restoration, which can lead to computational bottlenecks. In contrast, this paper eliminates such floating-point dependencies by converting all operations, including SoftMax, layer normalization and activation functions, into fixed-point integer formats. Furthermore, to maintain the accuracy in the integer computation, we partition the model’s computational graph into repeatable and one-to-one nodes (RONs) and hierarchically determine integer and fractional bit-widths, ensuring that the pre-trained parameters and the bit-width of constants and initial values used in inference are optimized. Experimental results show that the proposed approach maintains the same accuracy as the FP16/FP32 baseline while achieving up to a 84.67% reduction in hardware resource usage and approximately 16× inference speed-up, as analyzed using the Synopsys Design Compiler. This demonstrates that fully integer computation of LLMs can simultaneously achieve high performance and efficiency.