Estimating the quality of academic books from their descriptions with ChatGPT

IF 2.5 3区管理学 Q2 INFORMATION SCIENCE & LIBRARY SCIENCE

Journal of Academic Librarianship Pub Date : 2025-02-13 DOI:10.1016/j.acalib.2025.103023

Mike Thelwall, Andrew Cox

{"title":"Estimating the quality of academic books from their descriptions with ChatGPT","authors":"Mike Thelwall, Andrew Cox","doi":"10.1016/j.acalib.2025.103023","DOIUrl":null,"url":null,"abstract":"<div><div>Although indicators based on scholarly citations are widely used to support the evaluation of academic journals, alternatives are needed for scholarly book acquisitions. This article assesses the value of research quality scores from ChatGPT 4o-mini for 9830 social sciences, arts, and humanities books from 2019 indexed in Scopus, based on their titles and descriptions but not their full texts. Although most books scored the same (3* on a 1* to 4* scale), the citation rates correlate positively but weakly with ChatGPT 4o-mini research quality scores in both the social sciences and the arts and humanities. Part of the reason for the differences was the inclusion of textbooks, short books, and edited collections, all of which tended to be less cited and lower scoring. Some topics also tend to attract many/few citations and/or high/low ChatGPT scores. Descriptions explicitly mentioning theory and/or some methods also associated with higher scores and more citations. Overall, the results provide some evidence that both ChatGPT scores and citation counts are weak indicators of the research quality of books. Whilst not strong enough to support individual book quality judgements, they may help academic librarians seeking to evaluate new book collections, series, or publishers for potential acquisition.</div></div>","PeriodicalId":47762,"journal":{"name":"Journal of Academic Librarianship","volume":"51 2","pages":"Article 103023"},"PeriodicalIF":2.5000,"publicationDate":"2025-02-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Academic Librarianship","FirstCategoryId":"91","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0099133325000199","RegionNum":3,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"INFORMATION SCIENCE & LIBRARY SCIENCE","Score":null,"Total":0}

引用次数: 0

Abstract

Although indicators based on scholarly citations are widely used to support the evaluation of academic journals, alternatives are needed for scholarly book acquisitions. This article assesses the value of research quality scores from ChatGPT 4o-mini for 9830 social sciences, arts, and humanities books from 2019 indexed in Scopus, based on their titles and descriptions but not their full texts. Although most books scored the same (3* on a 1* to 4* scale), the citation rates correlate positively but weakly with ChatGPT 4o-mini research quality scores in both the social sciences and the arts and humanities. Part of the reason for the differences was the inclusion of textbooks, short books, and edited collections, all of which tended to be less cited and lower scoring. Some topics also tend to attract many/few citations and/or high/low ChatGPT scores. Descriptions explicitly mentioning theory and/or some methods also associated with higher scores and more citations. Overall, the results provide some evidence that both ChatGPT scores and citation counts are weak indicators of the research quality of books. Whilst not strong enough to support individual book quality judgements, they may help academic librarians seeking to evaluate new book collections, series, or publishers for potential acquisition.

查看原文本刊更多论文

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Academic Librarianship INFORMATION SCIENCE & LIBRARY SCIENCE-

CiteScore

5.30

自引率

15.40%

发文量

120

审稿时长

29 days

期刊介绍： The Journal of Academic Librarianship, an international and refereed journal, publishes articles that focus on problems and issues germane to college and university libraries. JAL provides a forum for authors to present research findings and, where applicable, their practical applications and significance; analyze policies, practices, issues, and trends; speculate about the future of academic librarianship; present analytical bibliographic essays and philosophical treatises. JAL also brings to the attention of its readers information about hundreds of new and recently published books in library and information science, management, scholarly communication, and higher education. JAL, in addition, covers management and discipline-based software and information policy developments.