QUBVIS：基于查询的多模态摘要系统，使用基于CLIP的转换器和视觉语言模型

IF 2.4 4区计算机科学 Q2 COMPUTER SCIENCE, SOFTWARE ENGINEERING

SoftwareX Pub Date : 2025-08-11 DOI:10.1016/j.softx.2025.102303

Turan Goktug Altundogan , Mehmet Karakose

{"title":"QUBVIS：基于查询的多模态摘要系统，使用基于CLIP的转换器和视觉语言模型","authors":"Turan Goktug Altundogan , Mehmet Karakose","doi":"10.1016/j.softx.2025.102303","DOIUrl":null,"url":null,"abstract":"<div><div>In this study, a new approach is proposed for user-interactive summarization of online videos. In the proposed approach, video-to-video summarization is performed with a very high success rate using a multimodal transformer architecture (QUBVIS) that also takes activity queries from the user as input, and the resulting summary video is subjected to captioning using a Vision Language Model with a GPT-2 decoder. The developed models are integrated with a Flask API and presented in a way that online video platforms can easily integrate into their systems. In addition, a simple web interface using this API is developed to provide API communication with the user. The performance evaluations of both models of the proposed method show our superiority over similar studies in the literature.</div></div>","PeriodicalId":21905,"journal":{"name":"SoftwareX","volume":"31 ","pages":"Article 102303"},"PeriodicalIF":2.4000,"publicationDate":"2025-08-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"QUBVIS: query based multi-modal summarization system using CLIP based transformer and vision language models\",\"authors\":\"Turan Goktug Altundogan , Mehmet Karakose\",\"doi\":\"10.1016/j.softx.2025.102303\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><div>In this study, a new approach is proposed for user-interactive summarization of online videos. In the proposed approach, video-to-video summarization is performed with a very high success rate using a multimodal transformer architecture (QUBVIS) that also takes activity queries from the user as input, and the resulting summary video is subjected to captioning using a Vision Language Model with a GPT-2 decoder. The developed models are integrated with a Flask API and presented in a way that online video platforms can easily integrate into their systems. In addition, a simple web interface using this API is developed to provide API communication with the user. The performance evaluations of both models of the proposed method show our superiority over similar studies in the literature.</div></div>\",\"PeriodicalId\":21905,\"journal\":{\"name\":\"SoftwareX\",\"volume\":\"31 \",\"pages\":\"Article 102303\"},\"PeriodicalIF\":2.4000,\"publicationDate\":\"2025-08-11\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"SoftwareX\",\"FirstCategoryId\":\"94\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S2352711025002699\",\"RegionNum\":4,\"RegionCategory\":\"计算机科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"COMPUTER SCIENCE, SOFTWARE ENGINEERING\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"SoftwareX","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2352711025002699","RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, SOFTWARE ENGINEERING","Score":null,"Total":0}

引用次数: 0

摘要

本研究提出了一种新的在线视频用户交互式摘要方法。在提出的方法中，视频到视频摘要使用多模态转换器架构（QUBVIS）以非常高的成功率执行，该架构还将用户的活动查询作为输入，并且使用带有GPT-2解码器的视觉语言模型对生成的摘要视频进行字幕处理。开发的模型与Flask API集成，并以在线视频平台可以轻松集成到其系统中的方式呈现。此外，使用该API开发了一个简单的web界面，以提供与用户的API通信。本文提出的方法的两个模型的性能评价表明我们优于文献中类似的研究。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

QUBVIS: query based multi-modal summarization system using CLIP based transformer and vision language models

In this study, a new approach is proposed for user-interactive summarization of online videos. In the proposed approach, video-to-video summarization is performed with a very high success rate using a multimodal transformer architecture (QUBVIS) that also takes activity queries from the user as input, and the resulting summary video is subjected to captioning using a Vision Language Model with a GPT-2 decoder. The developed models are integrated with a Flask API and presented in a way that online video platforms can easily integrate into their systems. In addition, a simple web interface using this API is developed to provide API communication with the user. The performance evaluations of both models of the proposed method show our superiority over similar studies in the literature.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

SoftwareX COMPUTER SCIENCE, SOFTWARE ENGINEERING-

CiteScore

5.50

自引率

2.90%

发文量

184

审稿时长

9 weeks

期刊介绍： SoftwareX aims to acknowledge the impact of software on today''s research practice, and on new scientific discoveries in almost all research domains. SoftwareX also aims to stress the importance of the software developers who are, in part, responsible for this impact. To this end, SoftwareX aims to support publication of research software in such a way that: The software is given a stamp of scientific relevance, and provided with a peer-reviewed recognition of scientific impact; The software developers are given the credits they deserve; The software is citable, allowing traditional metrics of scientific excellence to apply; The academic career paths of software developers are supported rather than hindered; The software is publicly available for inspection, validation, and re-use. Above all, SoftwareX aims to inform researchers about software applications, tools and libraries with a (proven) potential to impact the process of scientific discovery in various domains. The journal is multidisciplinary and accepts submissions from within and across subject domains such as those represented within the broad thematic areas below: Mathematical and Physical Sciences; Environmental Sciences; Medical and Biological Sciences; Humanities, Arts and Social Sciences. Originating from these broad thematic areas, the journal also welcomes submissions of software that works in cross cutting thematic areas, such as citizen science, cybersecurity, digital economy, energy, global resource stewardship, health and wellbeing, etcetera. SoftwareX specifically aims to accept submissions representing domain-independent software that may impact more than one research domain.