{"title":"Exploring the applicability of Large Language Models to citation context analysis","authors":"Kai Nishikawa, Hitoshi Koshiba","doi":"arxiv-2409.02443","DOIUrl":null,"url":null,"abstract":"Unlike traditional citation analysis -- which assumes that all citations in a\npaper are equivalent -- citation context analysis considers the contextual\ninformation of individual citations. However, citation context analysis\nrequires creating large amounts of data through annotation, which hinders the\nwidespread use of this methodology. This study explored the applicability of\nLarge Language Models (LLMs) -- particularly ChatGPT -- to citation context\nanalysis by comparing LLMs and human annotation results. The results show that\nthe LLMs annotation is as good as or better than the human annotation in terms\nof consistency but poor in terms of predictive performance. Thus, having LLMs\nimmediately replace human annotators in citation context analysis is\ninappropriate. However, the annotation results obtained by LLMs can be used as\nreference information when narrowing the annotation results obtained by\nmultiple human annotators to one, or LLMs can be used as one of the annotators\nwhen it is difficult to prepare sufficient human annotators. This study\nprovides basic findings important for the future development of citation\ncontext analyses.","PeriodicalId":501285,"journal":{"name":"arXiv - CS - Digital Libraries","volume":"65 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-09-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Digital Libraries","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.02443","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
Unlike traditional citation analysis -- which assumes that all citations in a
paper are equivalent -- citation context analysis considers the contextual
information of individual citations. However, citation context analysis
requires creating large amounts of data through annotation, which hinders the
widespread use of this methodology. This study explored the applicability of
Large Language Models (LLMs) -- particularly ChatGPT -- to citation context
analysis by comparing LLMs and human annotation results. The results show that
the LLMs annotation is as good as or better than the human annotation in terms
of consistency but poor in terms of predictive performance. Thus, having LLMs
immediately replace human annotators in citation context analysis is
inappropriate. However, the annotation results obtained by LLMs can be used as
reference information when narrowing the annotation results obtained by
multiple human annotators to one, or LLMs can be used as one of the annotators
when it is difficult to prepare sufficient human annotators. This study
provides basic findings important for the future development of citation
context analyses.