HTMLDownloader：一个使用WebView2进行动态网页抓取和存档的开源工具

IF 2.4 4区计算机科学 Q2 COMPUTER SCIENCE, SOFTWARE ENGINEERING

SoftwareX Pub Date : 2025-09-20 DOI:10.1016/j.softx.2025.102373

Ba-Vinh Truong , Loan T.T. Nguyen , Phu Pham , Bay Vo

{"title":"HTMLDownloader：一个使用WebView2进行动态网页抓取和存档的开源工具","authors":"Ba-Vinh Truong , Loan T.T. Nguyen , Phu Pham , Bay Vo","doi":"10.1016/j.softx.2025.102373","DOIUrl":null,"url":null,"abstract":"<div><div>The increasing complexity and dynamism of modern websites present major challenges for traditional web scraping tools such as Scrapy, BeautifulSoup and wget, which often fail to capture dynamic content or offer accessible user interfaces. To address these limitations, we introduce HTMLDownloader which provides a graphical interface (GUI) that makes it accessible to non-technical users and enhances scraping reliability by integrating browser-based rendering. Experimental evaluations on 75,516 links from 35 diverse domains demonstrate a 98.4% success rate, significantly outperforming Selenium (81.3%), Scrapy (56.6%), BeautifulSoup (34.5%) and wget (45.1%). These results confirm HTMLDownloader’s robustness and scalability, making it a powerful solution for dynamic content extraction and long-term web archiving. HTMLDownloader ships as end-user MSI/portable ZIP with documented workflows, enabling non-specialists to reproducibly archive JavaScript-heavy pages. A DOI-tagged release supports verification, reuse and citation (DOI: 10.5281/zenodo.16935169).</div></div>","PeriodicalId":21905,"journal":{"name":"SoftwareX","volume":"32 ","pages":"Article 102373"},"PeriodicalIF":2.4000,"publicationDate":"2025-09-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"HTMLDownloader: An open-source tool for dynamic web scraping and archiving using WebView2\",\"authors\":\"Ba-Vinh Truong , Loan T.T. Nguyen , Phu Pham , Bay Vo\",\"doi\":\"10.1016/j.softx.2025.102373\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><div>The increasing complexity and dynamism of modern websites present major challenges for traditional web scraping tools such as Scrapy, BeautifulSoup and wget, which often fail to capture dynamic content or offer accessible user interfaces. To address these limitations, we introduce HTMLDownloader which provides a graphical interface (GUI) that makes it accessible to non-technical users and enhances scraping reliability by integrating browser-based rendering. Experimental evaluations on 75,516 links from 35 diverse domains demonstrate a 98.4% success rate, significantly outperforming Selenium (81.3%), Scrapy (56.6%), BeautifulSoup (34.5%) and wget (45.1%). These results confirm HTMLDownloader’s robustness and scalability, making it a powerful solution for dynamic content extraction and long-term web archiving. HTMLDownloader ships as end-user MSI/portable ZIP with documented workflows, enabling non-specialists to reproducibly archive JavaScript-heavy pages. A DOI-tagged release supports verification, reuse and citation (DOI: 10.5281/zenodo.16935169).</div></div>\",\"PeriodicalId\":21905,\"journal\":{\"name\":\"SoftwareX\",\"volume\":\"32 \",\"pages\":\"Article 102373\"},\"PeriodicalIF\":2.4000,\"publicationDate\":\"2025-09-20\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"SoftwareX\",\"FirstCategoryId\":\"94\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S2352711025003395\",\"RegionNum\":4,\"RegionCategory\":\"计算机科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"COMPUTER SCIENCE, SOFTWARE ENGINEERING\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"SoftwareX","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2352711025003395","RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, SOFTWARE ENGINEERING","Score":null,"Total":0}

引用次数: 0

摘要

现代网站日益增加的复杂性和动态性对传统的网络抓取工具（如Scrapy、BeautifulSoup和wget）提出了重大挑战，这些工具通常无法捕获动态内容或提供可访问的用户界面。为了解决这些限制，我们引入了HTMLDownloader，它提供了一个图形界面（GUI），使非技术用户可以访问它，并通过集成基于浏览器的呈现来提高抓取的可靠性。对来自35个不同域名的75,516个链接的实验评估表明，成功率为98.4%，显著优于Selenium (81.3%), Scrapy (56.6%), BeautifulSoup（34.5%）和wget（45.1%）。这些结果证实了HTMLDownloader的健壮性和可扩展性，使其成为动态内容提取和长期web存档的强大解决方案。HTMLDownloader以终端用户MSI/便携ZIP的形式发布，带有文档化的工作流，使非专业人员能够重复地归档javascript繁重的页面。带有DOI标记的版本支持验证、重用和引用（DOI: 10.5281/zenodo.16935169）。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

HTMLDownloader: An open-source tool for dynamic web scraping and archiving using WebView2

The increasing complexity and dynamism of modern websites present major challenges for traditional web scraping tools such as Scrapy, BeautifulSoup and wget, which often fail to capture dynamic content or offer accessible user interfaces. To address these limitations, we introduce HTMLDownloader which provides a graphical interface (GUI) that makes it accessible to non-technical users and enhances scraping reliability by integrating browser-based rendering. Experimental evaluations on 75,516 links from 35 diverse domains demonstrate a 98.4% success rate, significantly outperforming Selenium (81.3%), Scrapy (56.6%), BeautifulSoup (34.5%) and wget (45.1%). These results confirm HTMLDownloader’s robustness and scalability, making it a powerful solution for dynamic content extraction and long-term web archiving. HTMLDownloader ships as end-user MSI/portable ZIP with documented workflows, enabling non-specialists to reproducibly archive JavaScript-heavy pages. A DOI-tagged release supports verification, reuse and citation (DOI: 10.5281/zenodo.16935169).

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

SoftwareX COMPUTER SCIENCE, SOFTWARE ENGINEERING-

CiteScore

5.50

自引率

2.90%

发文量

184

审稿时长

9 weeks

期刊介绍： SoftwareX aims to acknowledge the impact of software on today''s research practice, and on new scientific discoveries in almost all research domains. SoftwareX also aims to stress the importance of the software developers who are, in part, responsible for this impact. To this end, SoftwareX aims to support publication of research software in such a way that: The software is given a stamp of scientific relevance, and provided with a peer-reviewed recognition of scientific impact; The software developers are given the credits they deserve; The software is citable, allowing traditional metrics of scientific excellence to apply; The academic career paths of software developers are supported rather than hindered; The software is publicly available for inspection, validation, and re-use. Above all, SoftwareX aims to inform researchers about software applications, tools and libraries with a (proven) potential to impact the process of scientific discovery in various domains. The journal is multidisciplinary and accepts submissions from within and across subject domains such as those represented within the broad thematic areas below: Mathematical and Physical Sciences; Environmental Sciences; Medical and Biological Sciences; Humanities, Arts and Social Sciences. Originating from these broad thematic areas, the journal also welcomes submissions of software that works in cross cutting thematic areas, such as citizen science, cybersecurity, digital economy, energy, global resource stewardship, health and wellbeing, etcetera. SoftwareX specifically aims to accept submissions representing domain-independent software that may impact more than one research domain.