Reducing the effect of face orientation using FaceMesh landmarks in drowsiness estimation based on facial thermal images

Abstract

In this study, facial skin temperature distribution (FSTD) is focused on as a new driver monitoring index. FSTD is an autonomic index that can be measured remotely. Studies have been conducted to estimate drowsiness based on FSTD using modelng methods such as CNN, a type of deep learning, and sparse modeling, which can be trained with a small amount of data. These studies, however, only evaluated front-facing facial thermal images. FaceMesh is a model that extracts 478 3D facial feature landmarks from a 2D face image. In contrast to conventional models that extract only 68 facial feature landmarks, FaceMesh can extract facial feature landmarks for the entire face, including the cheeks, forehead, and other areas of the face that are in the blind spots. This study aims to improve the accuracy of drowsiness estimation by applying FaceMesh to automatically detect tilted faces and not including tilted images in the training data. As a result, the method proposed in this study improved drowsiness estimation accuracy by about 6% compared to the old method, which did not take face orientation into account.