Findings of ACL. NAACL最新文献

Tooling or Not Tooling? The Impact of Tools on Language Agents for Chemistry Problem Solving. 工具还是不工具？工具对化学问题解决语言代理的影响。

Findings of ACL. NAACL Pub Date : 2025-04-01 DOI: 10.18653/v1/2025.findings-naacl.424

Botao Yu, Frazier N Baker, Ziru Chen, Garrett Herb, Boyu Gou, Daniel Adu-Ampratwum, Xia Ning, Huan Sun

引用次数: 0

LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models. LMOD：大型多模态眼科学数据集和大型视觉语言模型的基准。

Findings of ACL. NAACL Pub Date : 2025-04-01 DOI: 10.18653/v1/2025.findings-naacl.135

Zhenyue Qin, Yu Yin, Dylan Campbell, Xuansheng Wu, Ke Zou, Yih-Chung Tham, Ninghao Liu, Xiuzhen Zhang, Qingyu Chen

{"title":"LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models.","authors":"Zhenyue Qin, Yu Yin, Dylan Campbell, Xuansheng Wu, Ke Zou, Yih-Chung Tham, Ninghao Liu, Xiuzhen Zhang, Qingyu Chen","doi":"10.18653/v1/2025.findings-naacl.135","DOIUrl":"10.18653/v1/2025.findings-naacl.135","url":null,"abstract":"The prevalence of vision-threatening eye diseases is a significant global burden, with many cases remaining undiagnosed or diagnosed too late for effective treatment. Large vision-language models (LVLMs) have the potential to assist in understanding anatomical information, diagnosing eye diseases, and drafting interpretations and follow-up plans, thereby reducing the burden on clinicians and improving access to eye care. However, limited benchmarks are available to assess LVLMs' performance in ophthalmology-specific applications. In this study, we introduce LMOD, a large-scale multimodal ophthalmology benchmark consisting of 21,993 instances across (1) five ophthalmic imaging modalities: optical coherence tomography, color fundus photographs, scanning laser ophthalmoscopy, lens photographs, and surgical scenes; (2) free-text, demographic, and disease biomarker information; and (3) primary ophthalmology-specific applications such as anatomical information understanding, disease diagnosis, and subgroup analysis. In addition, we benchmarked 13 state-of-the-art LVLM representatives from closed-source, open-source, and medical domains. The results demonstrate a significant performance drop for LVLMs in ophthalmology compared to other domains. Systematic error analysis further identified six major failure modes: misclassification, failure to abstain, inconsistent reasoning, hallucination, assertions without justification, and lack of domain-specific knowledge. In contrast, supervised neural networks specifically trained on these tasks as baselines demonstrated high accuracy. These findings underscore the pressing need for benchmarks in the development and validation of ophthalmology-specific LVLMs.","PeriodicalId":521134,"journal":{"name":"Findings of ACL. NAACL","volume":"2025 ","pages":"2501-2522"},"PeriodicalIF":0.0,"publicationDate":"2025-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12764179/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145902248","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Uncertainty Quantification for Clinical Outcome Predictions with (Large) Language Models. 用（大）语言模型进行临床结果预测的不确定性量化。

Findings of ACL. NAACL Pub Date : 2025-04-01 DOI: 10.18653/v1/2025.findings-naacl.419

Zizhang Chen, Peizhao Li, Xiaomeng Dong, Pengyu Hong

{"title":"Uncertainty Quantification for Clinical Outcome Predictions with (Large) Language Models.","authors":"Zizhang Chen, Peizhao Li, Xiaomeng Dong, Pengyu Hong","doi":"10.18653/v1/2025.findings-naacl.419","DOIUrl":"10.18653/v1/2025.findings-naacl.419","url":null,"abstract":"To facilitate healthcare delivery, language models (LMs) have significant potential for clinical prediction tasks using electronic health records (EHRs). However, in these high-stakes applications, unreliable decisions can result in high costs due to compromised patient safety and ethical concerns, thus increasing the need for good uncertainty modeling of automated clinical predictions. To address this, we consider uncertainty quantification of LMs for EHR tasks in both white-box and black-box settings. We first quantify uncertainty in white-box models, where we have access to model parameters and output logits. We show that an effective reduction of model uncertainty can be achieved by using the proposed multi-tasking and ensemble methods in EHRs. Continuing with this idea, we extend our approach to black-box settings, including popular proprietary LMs such as GPT-4. We validate our framework using longitudinal clinical data from over 6,000 patients across ten clinical prediction tasks. Results show that ensembling methods and multi-task prediction prompts reduce uncertainty across different scenarios. These findings increase model transparency in white-box and black-box settings, thereby advancing reliable AI healthcare. Our code is publically available at https://github.com/Cyrus9721/EHR_Uncertainty.","PeriodicalId":521134,"journal":{"name":"Findings of ACL. NAACL","volume":"2025 ","pages":"7512-7523"},"PeriodicalIF":0.0,"publicationDate":"2025-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12670956/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145673507","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Semantic Consistency-Based Uncertainty Quantification for Factuality in Radiology Report Generation. 基于语义一致性的放射学报告真实性不确定性量化。

Findings of ACL. NAACL Pub Date : 2025-04-01 DOI: 10.18653/v1/2025.findings-naacl.95

Chenyu Wang, Weichao Zhou, Shantanu Ghosh, Kayhan Batmanghelich, Wenchao Li

{"title":"Semantic Consistency-Based Uncertainty Quantification for Factuality in Radiology Report Generation.","authors":"Chenyu Wang, Weichao Zhou, Shantanu Ghosh, Kayhan Batmanghelich, Wenchao Li","doi":"10.18653/v1/2025.findings-naacl.95","DOIUrl":"10.18653/v1/2025.findings-naacl.95","url":null,"abstract":"Radiology report generation (RRG) has shown great potential in assisting radiologists by automating the labor-intensive task of report writing. While recent advancements have improved the quality and coherence of generated reports, ensuring their factual correctness remains a critical challenge. Although generative medical Vision Large Language Models (VLLMs) have been proposed to address this issue, these models are prone to hallucinations and can produce inaccurate diagnostic information. To address these concerns, we introduce a novel Semantic Consistency-Based Uncertainty Quantification framework that provides both report-level and sentence-level uncertainties. Unlike existing approaches, our method does not require modifications to the underlying model or access to its inner state, such as output token logits, thus serving as a plug-and-play module that can be seamlessly integrated with state-of-the-art models. Extensive experiments demonstrate the efficacy of our method in detecting hallucinations and enhancing the factual accuracy of automatically generated radiology reports. By abstaining from high-uncertainty reports, our approach improves factuality scores by 10%, achieved by rejecting 20% of reports using the Radialog model on the MIMIC-CXR dataset. Furthermore, sentence-level uncertainty flags the lowestprecision sentence in each report with an 82.9% success rate. Our implementation is open-source and available at https://github.com/BU-DEPEND-Lab/SCUQ-RRG.","PeriodicalId":521134,"journal":{"name":"Findings of ACL. NAACL","volume":"2025 ","pages":"1739-1754"},"PeriodicalIF":0.0,"publicationDate":"2025-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12756947/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145902196","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

OSCaR: Object State Captioning and State Change Representation. OSCaR：对象状态字幕和状态变化表示。

Findings of ACL. NAACL Pub Date : 2024-06-01 DOI: 10.18653/v1/2024.findings-naacl.226

Nguyen Nguyen, Jing Bi, Ali Vosoughi, Yapeng Tian, Pooyan Fazli, Chenliang Xu

{"title":"OSCaR: Object State Captioning and State Change Representation.","authors":"Nguyen Nguyen, Jing Bi, Ali Vosoughi, Yapeng Tian, Pooyan Fazli, Chenliang Xu","doi":"10.18653/v1/2024.findings-naacl.226","DOIUrl":"10.18653/v1/2024.findings-naacl.226","url":null,"abstract":"The capability of intelligent models to extrapolate and comprehend changes in object states is a crucial yet demanding aspect of AI research, particularly through the lens of human interaction in real-world settings. This task involves describing complex visual environments, identifying active objects, and interpreting their changes as conveyed through language. Traditional methods, which isolate object captioning and state change detection, offer a limited view of dynamic environments. Moreover, relying on a small set of symbolic words to represent changes has restricted the expressiveness of language. To address these challenges, in this paper, we introduce the Object State Captioning and State Change Representation (OSCaR) dataset and benchmark. OSCaR consists of 14,084 annotated video segments with nearly 1,000 unique objects from various egocentric video collections. It sets a new testbed for evaluating Multimodal Large Language Models (MLLMs). Our experiments demonstrate that while MLLMs show some skill, they lack a full understanding of object state changes. The benchmark includes a fine-tuned model that, despite initial capabilities, requires significant improvements in accuracy and generalization ability for effective understanding of these changes. Our code and dataset are available at https://github.com/nguyennm1024/OSCaR.","PeriodicalId":521134,"journal":{"name":"Findings of ACL. NAACL","volume":"2024 ","pages":"3565-3576"},"PeriodicalIF":0.0,"publicationDate":"2024-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12665500/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145663453","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Identifying Self-Disclosures of Use, Misuse and Addiction in Community-based Social Media Posts. 识别基于社区的社交媒体帖子中使用、滥用和成瘾的自我披露。

Findings of ACL. NAACL Pub Date : 2024-06-01 DOI: 10.18653/v1/2024.findings-naacl.161

Chenghao Yang, Tuhin Chakrabarty, Karli R Hochstatter, Melissa N Slavin, Nabila El-Bassel, Smaranda Muresan

{"title":"Identifying Self-Disclosures of Use, Misuse and Addiction in Community-based Social Media Posts.","authors":"Chenghao Yang, Tuhin Chakrabarty, Karli R Hochstatter, Melissa N Slavin, Nabila El-Bassel, Smaranda Muresan","doi":"10.18653/v1/2024.findings-naacl.161","DOIUrl":"10.18653/v1/2024.findings-naacl.161","url":null,"abstract":"In the last decade, the United States has lost more than 500,000 people from an overdose involving prescription and illicit opioids making it a national public health emergency (USDHHS, 2017). Medical practitioners require robust and timely tools that can effectively identify at-risk patients. Community-based social media platforms such as Reddit allow self-disclosure for users to discuss otherwise sensitive drug-related behaviors. We present a moderate size corpus of 2500 opioid-related posts from various subreddits labeled with six different phases of opioid use: Medical Use, Misuse, Addiction, Recovery, Relapse, Not Using. For every post, we annotate span-level extractive explanations and crucially study their role both in annotation quality and model development. We evaluate several state-of-the-art models in a supervised, few-shot, or zero-shot setting. Experimental results and error analysis show that identifying the phases of opioid use disorder is highly contextual and challenging. However, we find that using explanations during modeling leads to a significant boost in classification accuracy demonstrating their beneficial role in a high-stakes domain such as studying the opioid use disorder continuum.","PeriodicalId":521134,"journal":{"name":"Findings of ACL. NAACL","volume":"2024 ","pages":"2507-2521"},"PeriodicalIF":0.0,"publicationDate":"2024-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12671919/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145673506","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0