Photogrammetry engaged automated image labeling approach

Abstract

Deep learning models require many instances of training data to be able to accurately detect the desired object. However, the labeling of images is currently conducted manually due to the inclusion of irrelevant scenes in the original images, especially for the data collected in a dynamic environment such as from drone imagery. In this work, we developed an automated extraction of training data set using photogrammetry. This approach works with continuous and arbitrary collection of visual data, such as video, encompassing a stationary object. A dense point cloud was first generated to estimate the geometric relationship between individual images using a structure-from-motion (SfM) technique, followed by user-designated region-of-interests, ROIs, that are automatically extracted from the original images. An orthophoto mosaic of the façade plane of the building shown in the point cloud was created to ease the user’s selection of an intended labeling region of the object, which is a one-time process. We verified this method by using the ROIs extracted from a previously obtained dataset to train and test a convolutional neural network which is modeled to detect damage locations. The method put forward in this work allows a relatively small amount of labeling to generate a large amount of training data. We successfully demonstrate the capabilities of the technique with the dataset previously collected by a drone from an abandoned building in which many of the glass windows have been damaged.