Uncertainty in Visual Generative AI

IF 16.4 1区化学 Q1 CHEMISTRY, MULTIDISCIPLINARY

Accounts of Chemical Research Pub Date : 2024-03-27 DOI:10.3390/a17040136

Kara Combs, Adam Moyer, Trevor Bihl

{"title":"Uncertainty in Visual Generative AI","authors":"Kara Combs, Adam Moyer, Trevor Bihl","doi":"10.3390/a17040136","DOIUrl":null,"url":null,"abstract":"Recently, generative artificial intelligence (GAI) has impressed the world with its ability to create text, images, and videos. However, there are still areas in which GAI produces undesirable or unintended results due to being “uncertain”. Before wider use of AI-generated content, it is important to identify concepts where GAI is uncertain to ensure the usage thereof is ethical and to direct efforts for improvement. This study proposes a general pipeline to automatically quantify uncertainty within GAI. To measure uncertainty, the textual prompt to a text-to-image model is compared to captions supplied by four image-to-text models (GIT, BLIP, BLIP-2, and InstructBLIP). Its evaluation is based on machine translation metrics (BLEU, ROUGE, METEOR, and SPICE) and word embedding’s cosine similarity (Word2Vec, GloVe, FastText, DistilRoBERTa, MiniLM-6, and MiniLM-12). The generative AI models performed consistently across the metrics; however, the vector space models yielded the highest average similarity, close to 80%, which suggests more ideal and “certain” results. Suggested future work includes identifying metrics that best align with a human baseline to ensure quality and consideration for more GAI models. The work within can be used to automatically identify concepts in which GAI is “uncertain” to drive research aimed at increasing confidence in these areas.","PeriodicalId":1,"journal":{"name":"Accounts of Chemical Research","volume":"46 12","pages":""},"PeriodicalIF":16.4000,"publicationDate":"2024-03-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Accounts of Chemical Research","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.3390/a17040136","RegionNum":1,"RegionCategory":"化学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"CHEMISTRY, MULTIDISCIPLINARY","Score":null,"Total":0}

引用次数: 0

Abstract

Recently, generative artificial intelligence (GAI) has impressed the world with its ability to create text, images, and videos. However, there are still areas in which GAI produces undesirable or unintended results due to being “uncertain”. Before wider use of AI-generated content, it is important to identify concepts where GAI is uncertain to ensure the usage thereof is ethical and to direct efforts for improvement. This study proposes a general pipeline to automatically quantify uncertainty within GAI. To measure uncertainty, the textual prompt to a text-to-image model is compared to captions supplied by four image-to-text models (GIT, BLIP, BLIP-2, and InstructBLIP). Its evaluation is based on machine translation metrics (BLEU, ROUGE, METEOR, and SPICE) and word embedding’s cosine similarity (Word2Vec, GloVe, FastText, DistilRoBERTa, MiniLM-6, and MiniLM-12). The generative AI models performed consistently across the metrics; however, the vector space models yielded the highest average similarity, close to 80%, which suggests more ideal and “certain” results. Suggested future work includes identifying metrics that best align with a human baseline to ensure quality and consideration for more GAI models. The work within can be used to automatically identify concepts in which GAI is “uncertain” to drive research aimed at increasing confidence in these areas.

查看原文本刊更多论文

视觉生成人工智能中的不确定性

最近，生成式人工智能（GAI）创造文本、图像和视频的能力给世人留下了深刻印象。然而，由于 "不确定性"，GAI 仍会在某些领域产生不理想或意外的结果。在更广泛地使用人工智能生成的内容之前，有必要确定 GAI 不确定的概念，以确保其使用符合道德规范，并指导改进工作。本研究提出了一种自动量化 GAI 中不确定性的通用方法。为了测量不确定性，将文本到图像模型的文本提示与四个图像到文本模型（GIT、BLIP、BLIP-2 和 InstructBLIP）提供的标题进行比较。其评估基于机器翻译指标（BLEU、ROUGE、METEOR 和 SPICE）和词嵌入的余弦相似度（Word2Vec、GloVe、FastText、DistilRoBERTa、MiniLM-6 和 MiniLM-12）。生成式人工智能模型在各项指标中的表现一致；然而，向量空间模型产生的平均相似度最高，接近 80%，这表明结果更加理想和 "确定"。建议今后开展的工作包括确定与人类基线最匹配的指标，以确保质量并考虑更多的 GAI 模型。这些工作可用于自动识别 GAI 存在 "不确定性 "的概念，以推动旨在提高这些领域可信度的研究。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Accounts of Chemical Research 化学-化学综合

CiteScore

31.40

自引率

1.10%

发文量

312

审稿时长

2 months

期刊介绍： Accounts of Chemical Research presents short, concise and critical articles offering easy-to-read overviews of basic research and applications in all areas of chemistry and biochemistry. These short reviews focus on research from the author’s own laboratory and are designed to teach the reader about a research project. In addition, Accounts of Chemical Research publishes commentaries that give an informed opinion on a current research problem. Special Issues online are devoted to a single topic of unusual activity and significance. Accounts of Chemical Research replaces the traditional article abstract with an article "Conspectus." These entries synopsize the research affording the reader a closer look at the content and significance of an article. Through this provision of a more detailed description of the article contents, the Conspectus enhances the article's discoverability by search engines and the exposure for the research.