Visibly Pushdown Languages for a GUI Parsing Application with Probabilistic Lexer

Automatic understanding of GUI (Graphic User Interfaces) is vitally important for applications such as quality assurance, user monitoring, speech activated devices, automatic generation of GUI for application accessibility, and GUI design. Likewise, automatic understanding of visually structured documents (e.g. PDF files) is vitally important for data mining purposes. Current GUI parsers share two major shortcomings: First, instead of representing the user experience, they are tightly coupled to the underlying object model of the GUI. Second, from a linguistic point of view, they are either too restrictive to describe enough GUIs, or too permissive, in which case, the language structure itself becomes very fragile. We designed and implemented a new GUI parsing language which avoids these problems. It is easy to maintain, robust to changes in the input, and finally - as a computer program - decidable and fast to parse.