A deep learning approach for cultural heritage building classification using transfer learning and data augmentation

Abstract

The detection of architectural components in historic buildings is essential for digital documentation and the conservation process of cultural heritage. In this regard, recent studies have explored artificial intelligence with computer vision to enhance the detection of key components in monuments. However, this field of research still lacks investigation into the influence of using transfer learning and data augmentation to improve the performance of machine learning models. Moreover, the literature still requires research on Artificial Intelligence applied to Brazilian colonial architecture. Thus, this study proposes a new deep learning approach for cultural heritage building classification using transfer learning and data augmentation. For this purpose, the ImageMG dataset is proposed, containing 6449 images of 94 historic buildings from the state of Minas Gerais (Brazil), categorized into five classes: fronton, church, door, window, and tower. Additionally, the influence of using transfer learning to enhance the classification results of the Mobilenet architecture in the task of detecting components of historic buildings is evaluated. The proposed approach also investigates the effects of 64 combinations of data augmentation, utilizing six geometric transformations (zoom, width shift range, height shift range, vertical flip, horizontal flip, and rotation) for generating synthetic images to train the deep learning models. The results showed that the optimization of transfer learning in conjunction with data augmentation demonstrated significant advances in the performance of cultural heritage building classification. Experiments with the ImageMG dataset using transfer learning and vertical flip achieved the best accuracy results in validation (92.37 %), test 1 (90.22 %), and test 2 (87.33 %).