大型语言模型如何理解材料科学中的表格?

IF 2.4 3区 材料科学 Q3 ENGINEERING, MANUFACTURING
Defne Circi, Ghazal Khalighinejad, Anlan Chen, Bhuwan Dhingra, L. Catherine Brinson
{"title":"大型语言模型如何理解材料科学中的表格?","authors":"Defne Circi, Ghazal Khalighinejad, Anlan Chen, Bhuwan Dhingra, L. Catherine Brinson","doi":"10.1007/s40192-024-00362-6","DOIUrl":null,"url":null,"abstract":"<p>Advances in materials science require leveraging past findings and data from the vast published literature. While some materials data repositories are being built, they typically rely on newly created data in narrow domains because extracting detailed data and metadata from the enormous wealth of publications is immensely challenging. The advent of large language models (LLMs) presents a new opportunity to rapidly and accurately extract data and insights from the published literature and transform it into structured data formats for easy query and reuse. In this paper, we build on initial strategies for using LLMs for rapid and autonomous data extraction from materials science articles in a format curatable by materials databases. We presented the subdomain of polymer composites as our example use case and demonstrated the success and challenges of LLMs on extracting tabular data. We explored different table representations for use with LLMs, finding that a multimodal model with an image input yielded the most promising results. This model achieved an accuracy score of 0.910 for composition information extraction and an F<span>\\(_1\\)</span> score of 0.863 for property name information extraction. With the most conservative evaluation for the property extraction requiring exact match in all the details, we obtained an F<span>\\(_1\\)</span> score of 0.419. We observed that by allowing varying degrees of flexibility in the evaluation, the score can increase to 0.769. We envision that the results and analysis from this study will promote further research directions in developing information extraction strategies from materials information sources.</p>","PeriodicalId":13604,"journal":{"name":"Integrating Materials and Manufacturing Innovation","volume":null,"pages":null},"PeriodicalIF":2.4000,"publicationDate":"2024-07-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"How Well Do Large Language Models Understand Tables in Materials Science?\",\"authors\":\"Defne Circi, Ghazal Khalighinejad, Anlan Chen, Bhuwan Dhingra, L. Catherine Brinson\",\"doi\":\"10.1007/s40192-024-00362-6\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p>Advances in materials science require leveraging past findings and data from the vast published literature. While some materials data repositories are being built, they typically rely on newly created data in narrow domains because extracting detailed data and metadata from the enormous wealth of publications is immensely challenging. The advent of large language models (LLMs) presents a new opportunity to rapidly and accurately extract data and insights from the published literature and transform it into structured data formats for easy query and reuse. In this paper, we build on initial strategies for using LLMs for rapid and autonomous data extraction from materials science articles in a format curatable by materials databases. We presented the subdomain of polymer composites as our example use case and demonstrated the success and challenges of LLMs on extracting tabular data. We explored different table representations for use with LLMs, finding that a multimodal model with an image input yielded the most promising results. This model achieved an accuracy score of 0.910 for composition information extraction and an F<span>\\\\(_1\\\\)</span> score of 0.863 for property name information extraction. With the most conservative evaluation for the property extraction requiring exact match in all the details, we obtained an F<span>\\\\(_1\\\\)</span> score of 0.419. We observed that by allowing varying degrees of flexibility in the evaluation, the score can increase to 0.769. We envision that the results and analysis from this study will promote further research directions in developing information extraction strategies from materials information sources.</p>\",\"PeriodicalId\":13604,\"journal\":{\"name\":\"Integrating Materials and Manufacturing Innovation\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":2.4000,\"publicationDate\":\"2024-07-19\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Integrating Materials and Manufacturing Innovation\",\"FirstCategoryId\":\"88\",\"ListUrlMain\":\"https://doi.org/10.1007/s40192-024-00362-6\",\"RegionNum\":3,\"RegionCategory\":\"材料科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q3\",\"JCRName\":\"ENGINEERING, MANUFACTURING\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Integrating Materials and Manufacturing Innovation","FirstCategoryId":"88","ListUrlMain":"https://doi.org/10.1007/s40192-024-00362-6","RegionNum":3,"RegionCategory":"材料科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"ENGINEERING, MANUFACTURING","Score":null,"Total":0}
引用次数: 0

摘要

材料科学的进步需要利用过去从大量出版文献中获得的发现和数据。虽然目前正在建立一些材料数据资源库,但它们通常依赖于狭窄领域的新创建数据,因为从大量出版物中提取详细数据和元数据是一项巨大的挑战。大型语言模型(LLM)的出现提供了一个新的机会,可以快速、准确地从已发表的文献中提取数据和见解,并将其转换为结构化数据格式,以便于查询和重用。在本文中,我们在使用 LLMs 从材料科学文章中快速、自主地提取数据的初步策略基础上,将其转化为材料数据库可处理的格式。我们以聚合物复合材料子领域为例,展示了 LLMs 在提取表格数据方面的成功经验和面临的挑战。我们探索了与 LLM 配合使用的不同表格表示法,结果发现,使用图像输入的多模态模型取得了最理想的结果。该模型在成分信息提取方面的准确率达到了 0.910,在属性名称信息提取方面的准确率达到了 0.863。在要求所有细节完全匹配的最保守的属性提取评估中,我们得到的 F\(_1\) 分数为 0.419。我们观察到,如果在评估中允许不同程度的灵活性,得分可以提高到 0.769。我们希望本研究的结果和分析能进一步推动从材料信息源中开发信息提取策略的研究方向。
本文章由计算机程序翻译,如有差异,请以英文原文为准。

How Well Do Large Language Models Understand Tables in Materials Science?

How Well Do Large Language Models Understand Tables in Materials Science?

Advances in materials science require leveraging past findings and data from the vast published literature. While some materials data repositories are being built, they typically rely on newly created data in narrow domains because extracting detailed data and metadata from the enormous wealth of publications is immensely challenging. The advent of large language models (LLMs) presents a new opportunity to rapidly and accurately extract data and insights from the published literature and transform it into structured data formats for easy query and reuse. In this paper, we build on initial strategies for using LLMs for rapid and autonomous data extraction from materials science articles in a format curatable by materials databases. We presented the subdomain of polymer composites as our example use case and demonstrated the success and challenges of LLMs on extracting tabular data. We explored different table representations for use with LLMs, finding that a multimodal model with an image input yielded the most promising results. This model achieved an accuracy score of 0.910 for composition information extraction and an F\(_1\) score of 0.863 for property name information extraction. With the most conservative evaluation for the property extraction requiring exact match in all the details, we obtained an F\(_1\) score of 0.419. We observed that by allowing varying degrees of flexibility in the evaluation, the score can increase to 0.769. We envision that the results and analysis from this study will promote further research directions in developing information extraction strategies from materials information sources.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Integrating Materials and Manufacturing Innovation
Integrating Materials and Manufacturing Innovation Engineering-Industrial and Manufacturing Engineering
CiteScore
5.30
自引率
9.10%
发文量
42
审稿时长
39 days
期刊介绍: The journal will publish: Research that supports building a model-based definition of materials and processes that is compatible with model-based engineering design processes and multidisciplinary design optimization; Descriptions of novel experimental or computational tools or data analysis techniques, and their application, that are to be used for ICME; Best practices in verification and validation of computational tools, sensitivity analysis, uncertainty quantification, and data management, as well as standards and protocols for software integration and exchange of data; In-depth descriptions of data, databases, and database tools; Detailed case studies on efforts, and their impact, that integrate experiment and computation to solve an enduring engineering problem in materials and manufacturing.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信