Deep Information Retrieval Using Neural Language Models for Accounting Research: An Illustration Using Press News

Abstract

In recent years, machine learning (ML) models have been increasingly used to collect and analyze information from diverse sources, such as earning calls, news press releases, and social media. This has empowered accounting researchers to generate valuable insights in support of financial accounting, auditing, and corporate governance research. Compared to intensive manual reviews, ML models can automatically process millions of documents in days and generate statistical analyses. To incorporate text nuances into consideration, this paper demonstrates the application of large language models (LLMs) or fine-tuning a language model in extracting accounting-related information, using public web news announcements as an example of potential information sources. The proposed method is then evaluated using the latest well-known LLM services including ChatGPT, Llama and Sonnet, as well as the traditional probabilistic model. Additionally, this study discusses the advantages and disadvantages of different LLM models from the perspective of mismatches. As the standard ML benchmarking approach—human ground truth labeling through millions of pages—is time-consuming and expensive, it limits researchers' ability to acquire relevant information. Therefore, we compare mismatches in this study as an effective alternative approach to evaluate LLMs performances. This study contributes to the accounting literature by comparing the pros and cons of different LLMs in the context of extracting relevant information for accounting research. Moreover, it highlights the advantages of LLMs in understanding text nuances compared to traditional methods that rely on keywords and word frequency, helping accounting researchers select the appropriate language models for their work.