WHY AND HOW TO CAPTURE THE SEMANTICS OF WEB USER INTERFACES

Structured data on the Web has become very important over the last few years. Schema.org has proven to be the most recognized vocabulary for structuring data on the Web. However, most annotations using Schema.org's vocabulary describe exclusively content on websites. Today, these structured content data are mainly consumed by search engines to automatically browse websites with high speed and accuracy and to take over search efforts for humans. Annotated web user interfaces can hardly be found on websites or applications. Schema.org’s offered vocabulary to describe actions in a structured way plays a minor role on websites or applications and is mainly applied on rich emails. Also most structured data tools and platforms do not support the annotation of actions triggered via the user interface of a website or application. In this paper we motivate the importance of a semantic description of web user interfaces and present possible application areas to consume these structured data. Thereon, we investigate the suitability of Schema.org for the semantic enrichment of web user interfaces. Since there are hundreds of classes and properties in Schema.org, we present a conceptual model of Schema.org classes and properties suitable to annotate user interfaces of websites or applications. We also show to what extent interactive elements on HTML5 websites or applications can be automatically mapped to these model elements. As a proof of concept we introduce Schemator , our WYSIWYG tool to support the semantic enrichment process of web user interfaces. Schemator is not only a comfortable structured data tool but acts also as a SAAS platform to store and manage structured web interface data separated from the underlying websites and applications.