MorphGUI: Real-time GUIs customization with large language models

Abstract

Graphical user interface (GUI) customization relies on predefined configuration options and settings, constraining diverse individual needs and preferences within predetermined boundaries and often requiring technical expertise. To address these limitations, this work introduces MorphGUI, a framework leveraging Large Language Models (LLMs) to enable interface customization through natural language. By allowing users to express desired changes using their own words and harnessing the generative capabilities of LLMs, MorphGUI mitigates the limitations of predefined options and reduces the need for technical expertise. The framework translates functional and stylistic requests into either modifications of existing application components or generation of new ones. Through a use case implementation with a calendar application and a user study (n=18), where participants were tasked with modifying interfaces towards a target goal, we investigate whether MorphGUI can enable effective natural language-driven interface customization for non-expert users through both functional and visual modifications. Results show that participants successfully customized interfaces using natural language. Users found the system intuitive and achieved good performance regardless of technical background. We report analysis of optimal prompt length, challenges in separating functional and visual instructions in structured templates, correlation between LLM experience and success, and learning effects. The study revealed opportunities for enhanced guidance, examples, and scaffolding to help users structure their customization requests more effectively.