SynZ: Enhanced Synthetic Dataset for Training UI Element Detectors

Vinoth Pandian Sermuga Pandian, Sarah Suleri, M. Jarke
{"title":"SynZ: Enhanced Synthetic Dataset for Training UI Element Detectors","authors":"Vinoth Pandian Sermuga Pandian, Sarah Suleri, M. Jarke","doi":"10.1145/3397482.3450725","DOIUrl":null,"url":null,"abstract":"User Interface (UI) prototyping is an iterative process where designers initially sketch UIs before transforming them into interactive digital designs. Recent research applies Deep Neural Networks (DNNs) to identify the constituent UI elements of these UI sketches and transform these sketches into front-end code. Training such DNN models requires a large-scale dataset of UI sketches, which is time-consuming and expensive to collect. Therefore, we earlier proposed Syn to generate UI sketches synthetically by random allocation of UI element sketches. However, these UI sketches are not statistically similar to real-life UI screens. To bridge this gap, in this paper, we introduce the SynZ dataset, which contains 175,377 synthetically generated UI sketches statistically similar to real-life UI screens. To generate SynZ, we analyzed, enhanced, and extracted annotations from the RICO dataset and used 17,979 hand-drawn UI element sketches from the UISketch dataset. Further, we fine-tuned a UI element detector with SynZ and observed that it doubles the mean Average Precision of UI element detection compared to the Syn dataset.","PeriodicalId":216190,"journal":{"name":"26th International Conference on Intelligent User Interfaces - Companion","volume":"51 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2021-04-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"4","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"26th International Conference on Intelligent User Interfaces - Companion","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1145/3397482.3450725","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 4

Abstract

User Interface (UI) prototyping is an iterative process where designers initially sketch UIs before transforming them into interactive digital designs. Recent research applies Deep Neural Networks (DNNs) to identify the constituent UI elements of these UI sketches and transform these sketches into front-end code. Training such DNN models requires a large-scale dataset of UI sketches, which is time-consuming and expensive to collect. Therefore, we earlier proposed Syn to generate UI sketches synthetically by random allocation of UI element sketches. However, these UI sketches are not statistically similar to real-life UI screens. To bridge this gap, in this paper, we introduce the SynZ dataset, which contains 175,377 synthetically generated UI sketches statistically similar to real-life UI screens. To generate SynZ, we analyzed, enhanced, and extracted annotations from the RICO dataset and used 17,979 hand-drawn UI element sketches from the UISketch dataset. Further, we fine-tuned a UI element detector with SynZ and observed that it doubles the mean Average Precision of UI element detection compared to the Syn dataset.
用于训练UI元素检测器的增强合成数据集
用户界面(UI)原型设计是一个迭代过程,设计师在将其转换为交互式数字设计之前,首先绘制UI草图。最近的研究应用深度神经网络(dnn)来识别这些UI草图的组成元素,并将这些草图转换为前端代码。训练这样的DNN模型需要一个大规模的UI草图数据集,这是耗时和昂贵的收集。因此,我们早前提出Syn通过随机分配UI元素草图来综合生成UI草图。然而,这些UI草图在统计上与真实的UI屏幕并不相似。为了弥补这一差距,在本文中,我们引入了SynZ数据集,其中包含175,377个合成生成的UI草图,统计上与现实生活中的UI屏幕相似。为了生成SynZ,我们从RICO数据集中分析、增强和提取注释,并使用了来自usisketch数据集的17,979个手绘UI元素草图。此外,我们使用SynZ对UI元素检测器进行了微调,并观察到与Syn数据集相比,它将UI元素检测的平均精度提高了一倍。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信