Tools for Creating UI Automation Macros

Abstract

Automation macros enable users to perform digital tasks programmatically to save time or support hands-free interaction. For example, macros can be used to perform web scraping for a research project (e.g., scraping articles from a news site) or personal task automation via natural language (e.g., ordering food for delivery). Some kinds of automation are built into our devices and are readily available (e.g., via Siri [1] or Alexa [2]), but this set is limited and often will not support a user’s niche or complex needs. Users can create their own custom macros, but traditionally this requires writing program code which involves a significant amount of effort for programmers and is infeasible for non-programmers. In my PhD work, I have studied and designed tools for developers and end-users to more intuitively create user interface (UI) automation macros. First, I studied the challenges and needs of programmers writing web automation scripts [3], both in a traditional text editor and in richer environments, including a prototype IDE I built that provides context about the target UI and feedback on element selection logic. Next, I designed a programming by demonstration (PBD) environment that enables end-users to create custom macros for answering questions on the web without needing to write code [4]. So far my work has focused on helping users create macros that perform a desired web scraping or automation task on a single website. However, it makes sense that users might also want to perform the same kinds of automation on semantically similar websites. For example, a user might want to create an automation macro that can order food not only from the DoorDash website, but also from the GrubHub and Uber Eats websites. Another user might want to create an automation macro that scrapes data from staff directories across different department and university websites. In both cases, the websites’ content will be similar and the macro should perform the same high-level actions, but the websites’ exact visual appearance, widgets, and underlying implementation will vary. Currently users would need to create separate macros from scratch for each website. In my future work, I plan to help users create semantic macros -– macros that are capable of performing a given highlevel semantic task across different websites – without needing to create new automation logic from scratch for each new website. II. DEVELOPER ENVIRONMENTS FOR CREATING MACROS