DropMismatch: removing mismatched UI elements for better pixel to code generation

Automating the generation of user interface (UI) code from design images has gained significant attention due to its potential to streamline application development. However, the effectiveness of deep learning models in this domain is often hindered by mismatches between UI images and their corresponding layout code, a common issue in image-text datasets. In this paper, we introduce a framework that locates and removes these mismatches, thereby improving the accuracy of UI code generation models. Our approach leverages a convolutional neural network to predict the alignment between UI components and layout code nodes, coupled with a tree-based heuristic algorithm to localize mismatches. Through extensive evaluation, we demonstrate that our method enhances the accuracy of UI code generation by approximately 15%, while significantly reducing the need for costly manual annotations. The proposed framework not only advances the state of automated UI code generation but also lays the foundation for creating high-quality, large-scale UI datasets, essential for future research and development in this field.